A motion control method of a small bionic mouse robot based on reinforcement learning

Abstract

The application relates to a motion control method of a small bionic mouse robot based on reinforcement learning. First, an information processing flow based on minimum effective information extraction is constructed according to the structural characteristics of the bionic mouse robot, and basic elements in the reinforcement learning process are defined, including a state S, a reward function R and an action space A; the information processing based on minimum effective information extraction is defined as a network-based control decision process; in an initial simple scene without obstacles, a basic walking gait of the robot mouse is trained through a learning iteration process; finally, based on the training parameters in the scene without obstacles, the robot mouse is continuously trained in various different obstacle scenes to obtain an adaptive gait for crossing obstacles. The application designs a state space, an action space and a reward function according to the structural characteristics of the small mouse quadruped robot, and has better control effect; the learning iteration algorithm adopts an end-to-end control mode, and is simpler and better in dealing with complex terrains.

Description

Technical Field

[0001] This invention relates to the field of quadruped robot motion control technology, and more specifically, to a motion control method for a small bionic mouse robot based on reinforcement learning. Background Technology

[0002] Quadruped robots are widely used in various scenarios, making efficient gait control and adaptation to different terrains crucial for motion control. Currently, two common approaches are model-based computational control and reinforcement learning-based methods. However, model-based computational methods suffer from limited behavioral patterns and poor adaptability; existing reinforcement learning methods are mostly based on large rigid robots. Small, flexible robots have far fewer configurable resources than large robots, making it impossible to directly apply traditional reinforcement learning methods.

[0003] An existing patent describes a path planning method for quadruped robots based on DRL. Firstly, it utilizes the quadruped robot's built-in RGB... A D-camera pre-scans the environment, and a raster-based model of the motion environment is created to obtain an environmental map. This map is then decomposed into a series of raster units with binary information to obtain the initial state, which is used as input to a deep convolutional neural network. An attention mechanism is introduced to optimize the network structure to address the problem that the deep reinforcement learning DQN algorithm fails to effectively utilize key local information. Through end-to-end training, a reward function is set based on the movement characteristics of the quadruped robot, and the neural network is trained using a confidence interval upper bound exploration strategy to obtain the Q-values ​​of the eight actions that the quadruped robot can perform. Finally, an artificial potential field algorithm is used to continuously detect dynamic obstacles in the environment and perform real-time obstacle avoidance, achieving effective quadruped robot path planning. This method proposes a quadruped robot control method in complex terrain using information processing from visual sensors, achieving good results in path planning and obstacle crossing. However, due to the limited internal space and the limited number of sensors that can be mounted on small robots, they lack the ability to observe environmental image information, making this method difficult to use on small robots. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a motion control method for a small bionic mouse robot based on reinforcement learning, which has better control effect, is simpler to implement, and performs better when dealing with complex terrain.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A motion control method for a small biomimetic mouse robot based on reinforcement learning includes the following steps:

[0007] First, based on the structural characteristics of the bionic mouse robot, an information processing flow based on minimizing effective information extraction was constructed, and the basic elements of the reinforcement learning process were defined, including state S, reward function R, and action space A. Based on the information processing flow based on minimizing effective information extraction, a network-based control decision process was defined.

[0008] Next, in an initial simple, barrier-free scenario, the robot mouse is trained through an iterative learning process to acquire a basic walking gait. Finally, based on the training parameters in the barrier-free scenario, the robot mouse is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles.

[0009] In the specific control process, for any time t, the current state is obtained according to the definition of the state space. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter .

[0010] This invention proposes a motion control method for a small bionic quadruped mouse robot based on reinforcement learning, which mainly overcomes the shortcomings of existing reinforcement learning methods for quadruped robots, which are unsuitable for small robots. This control method designs a state space, action space, and reward function based on the structural characteristics of small mouse robots. It uses a learning iterative method based on the PPO algorithm to train the robot mouse to adaptively learn motion methods in various scenarios, enabling the small quadruped robot to autonomously learn how to walk and how to overcome various obstacles.

[0011] In one embodiment, the information processing flow based on minimizing the extraction of effective information includes:

[0012] S11. First, based on the structure of the robotic mouse, select the most simplified sensing data while meeting performance requirements; speed sensor readings. ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides; Represents the three-dimensional coordinates of the current position;

[0013] S12. Define the state space S = <V,Q, P q >;

[0014] S13. Calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ;

[0015] S14. Define the fall penalty function. The constant h is a height threshold used to determine whether the robot has fallen.

[0016] S15. Define the reward function based on cumulative displacement and fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target;

[0017] S16. Define the action space using an end-to-end approach:

[0018]

[0019] Each data component corresponds to the rotation angle of each servo motor of the robotic mouse.

[0020] In one embodiment, the process of decision control execution includes:

[0021] S21. Initialize the Actor network Through parameters definition ;

[0022] S22. Change the current state enter Obtain the probability distribution of the action space. ;

[0023] S23. On probability distribution Sampling to obtain action Select Action probability ;

[0024] S24. The robot performs the calculated action and receives a reward value from the environment. .

[0025] In one embodiment, the learning iteration process specifically includes:

[0026] S31. First, initialize the Actor network in an accessible scenario. parameter and Critic Network parameter ;

[0027] S32. Obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ;

[0028] S33. Will Add to experience buffer pool M;

[0029] S34. Repeat S32~S33 several times until the buffer pool M is full;

[0030] S35. Calculate the discounted reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M.

[0031] S36. Draw a mini-batch from buffer pool M for network training. , ;

[0032] S37. Repeat step S36 several times, then clear the experience buffer pool M;

[0033] S38. Repeat steps S32 to S37 several times until training is complete;

[0034] S39. Based on the parameters obtained after training , Repeat steps S32 to S38 under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

[0035] In one embodiment, step S36 specifically includes:

[0036] S361. Remove the mini-batch... Input network A new probability distribution is obtained and its cross-entropy ;

[0037] S362. Using a new probability distribution Calculate probability of occurrence ;

[0038] S363. [The following appears to be a separate, unrelated sentence:] Will Input network Get evaluation value Then the dominant value function ;

[0039] S364. Calculate the loss function ,in , , It is a parameter that indicates importance. Indicates the expected value. Indicates partial derivative; for and The mean square error, ,constant , ;

[0040] S365. Update using gradient descent with loss function parameters and parameters ,in This is the learning rate.

[0041] This invention also provides a motion control system for a small bionic mouse robot based on reinforcement learning, comprising:

[0042] Minimize effective information extraction module: Based on the structural characteristics of the bionic mouse robot, it constructs an information processing module based on minimizing effective information extraction, and defines the basic elements in the reinforcement learning process, including state S, reward function R, and action space A;

[0043] Decision Execution Module: Used to process the information stream extracted by the Minimize Effective Information Extraction Module and define the network-based control decision-making process;

[0044] The learning iteration module is used to train the robot mouse to acquire a basic walking gait in an initial, simple, barrier-free scenario through a learning iteration process. Finally, based on the training parameters in the barrier-free scenario, the robot mouse is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles.

[0045] In the specific control process, for any time t, the current state is obtained according to the state space definition. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter .

[0046] In one embodiment, the minimized effective information extraction module includes:

[0047] Sensor data extraction module: used to select the most concise sensor data based on the robotic mouse structure, while meeting performance requirements; speed sensor readings. ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides; Represents the three-dimensional coordinates of the current position;

[0048] State space definition module: used to define the state space S = <V,Q, P q >;

[0049] Cumulative Displacement Calculation Module: Used to calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ;

[0050] The fall penalty function definition module is used to define fall penalty functions. The constant h is a height threshold used to determine whether the robot has fallen.

[0051] The reward function definition module is used to define the reward function based on the cumulative displacement and the fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target;

[0052] Action space definition module: Used to define action space in an end-to-end manner.

[0053]

[0054] Each data component corresponds to the rotation angle of each servo motor of the robotic mouse.

[0055] In one embodiment, the decision execution module includes:

[0056] Initialization module: Used to initialize the Actor network. Through parameters definition ;

[0057] Action space probability distribution acquisition module: used to obtain the current state enter Obtain the probability distribution of the action space. ;

[0058] Action probability selection module: used for probability distribution Sampling to obtain action Select Action probability ;

[0059] Reward value acquisition module: Used by the robot to execute calculated actions and obtain reward values ​​from the environment. .

[0060] In one embodiment, the learning iteration module includes:

[0061] Network initialization module: Used to initialize the Actor network in accessible scenarios. parameter and Critic Network parameter ;

[0062] Selection module: Used to obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ;

[0063] Experience buffer pool addition module: used to add Add to experience buffer pool M;

[0064] First iteration module: Used to repeatedly run the selection module and the experience buffer pool addition module until the buffer pool M is full;

[0065] Discount Reward Calculation Module: Used to calculate the discount reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M.

[0066] Training module: Used to extract a mini-batch from buffer pool M for training the network. , ;

[0067] The second iteration module is used to repeatedly run the training module several times, and then clear the experience buffer pool M.

[0068] The third iteration module is used to repeatedly run the selection module, the experience buffer pool addition module, the first iteration module, the discount reward value calculation module, the training module, and the second iteration module several times until training is completed.

[0069] Agent parameter acquisition module: used to obtain parameters based on training. , The above modules are repeatedly run under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

[0070] In one embodiment, the training module includes:

[0071] Input module: Used to process data from mini-batch. Input network A new probability distribution is obtained and its cross-entropy ;

[0072] Probability calculation module: used to calculate new probability distributions Calculate probability of occurrence ;

[0073] Evaluation value module: used to... Input network Get evaluation value Then the dominant value function ;

[0074] Loss function calculation module: used to calculate the loss function. ,in , , It is a parameter that indicates importance. Indicates the expected value. Indicates partial derivative; for and The mean square error, ,constant , ;

[0075] Parameter update module: Used to update parameters using gradient descent based on the loss function. parameters and parameters ,in This is the learning rate.

[0076] Compared with existing technologies, the beneficial effects are as follows: The motion control method for a small bionic mouse robot based on reinforcement learning provided by this invention designs a state space, action space, and reward function specifically for the structural characteristics of small quadrupedal mouse robots, resulting in better control performance; the learning iteration algorithm of this invention adopts an end-to-end control method, which is simpler to implement and performs better when dealing with complex terrain. Attached Figure Description

[0077] Figure 1 A schematic diagram of the method flow of this invention.

[0078] Figure 2 This is a schematic diagram of the information processing flow based on minimizing the extraction of effective information according to the present invention.

[0079] Figure 3 This is a schematic diagram of the learning iteration process of this invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The present invention will be described in one embodiment below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only and represent schematic diagrams, not actual pictures, and should not be construed as limiting the present patent. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0081] In the description of this invention, it should be understood that if terms such as "upper," "lower," "left," and "right" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms describing positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting this patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances. In addition, if the embodiments of this invention involve descriptions of "first," "second," etc., such descriptions are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the meaning of "and / or" throughout the text is to include three parallel solutions. Taking "A and / or B" as an example, it includes solution A, or solution B, or a solution that simultaneously satisfies A and B.

[0082] Example 1:

[0083] This embodiment provides a motion control method for a small biomimetic mouse robot based on reinforcement learning, including the following steps:

[0084] First, based on the structural characteristics of the bionic mouse robot, an information processing flow based on minimizing effective information extraction was constructed, and the basic elements of the reinforcement learning process were defined, including state S, reward function R, and action space A; based on the information processing of minimizing effective information extraction, a network-based control decision process was defined.

[0085] Next, in an initial simple, barrier-free scenario, the robot mouse is trained through an iterative learning process to acquire a basic walking gait. Finally, based on the training parameters in the barrier-free scenario, the robot mouse is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles.

[0086] In the specific control process, for any time t, the current state is obtained according to the definition of the state space. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter .

[0087] This invention proposes a motion control method for a small bionic quadruped mouse robot based on reinforcement learning, which mainly overcomes the shortcomings of existing reinforcement learning methods for quadruped robots, which are unsuitable for small robots. This control method designs a state space, action space, and reward function based on the structural characteristics of small mouse robots. It uses a learning iterative method based on the PPO algorithm to train the robot mouse to adaptively learn motion methods in various scenarios, enabling the small quadruped robot to autonomously learn how to walk and how to overcome various obstacles.

[0088] The information processing flow based on minimizing the extraction of effective information includes:

[0089] S11. First, based on the structure of the robotic mouse, select the most simplified sensing data while meeting performance requirements; speed sensor readings. ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides; Represents the three-dimensional coordinates of the current position;

[0090] S12. Define the state space S = <V,Q,P q >;

[0091] S13. Calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ;

[0092] S14. Define the fall penalty function. The constant h is a height threshold used to determine whether the robot has fallen.

[0093] S15. Define the reward function based on cumulative displacement and fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target;

[0094] S16. Define the action space using an end-to-end approach:

[0095]

[0096] Each data component corresponds to the rotation angle of each servo motor of the robotic mouse.

[0097] Specifically, the process of decision control execution includes:

[0098] S21. Initialize the Actor network Through parameters definition ;

[0099] S22. Change the current state enter Obtain the probability distribution of the action space. ;

[0100] S23. On probability distribution Sampling to obtain action Select Action probability ;

[0101] S24. The robot performs the calculated action and receives a reward value from the environment. .

[0102] In addition, the learning iteration process specifically includes:

[0103] S31. First, initialize the Actor network in an accessible scenario. parameter and Critic Network parameter ;

[0104] S32. Obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ;

[0105] S33. Will Add to experience buffer pool M;

[0106] S34. Repeat S32~S33 several times until the buffer pool M is full;

[0107] S35. Calculate the discounted reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M.

[0108] S36. Draw a mini-batch from buffer pool M for network training. , ;

[0109] S37. Repeat step S6 several times, then clear the experience buffer pool M;

[0110] S38. Repeat steps S32 to S37 several times until training is complete;

[0111] S39. Based on the parameters obtained after training , Repeat steps S32 to S38 under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

[0112] Specifically, step S36 includes:

[0113] S361. Remove the mini-batch... Input network A new probability distribution is obtained and its cross-entropy ;

[0114] S362. Using a new probability distribution Calculate probability of occurrence ;

[0115] S363. [The following appears to be a separate, unrelated sentence:] Will Input network Get evaluation value Then the dominant value function ;

[0116] S364. Calculate the loss function ,in , , It is a parameter that indicates importance. for and The mean square error, Indicates the expected value. Indicates partial derivative, ,constant , ;

[0117] S365. Update using gradient descent with loss function parameters and parameters ,in This is the learning rate.

[0118] This invention designs a state space, action space, and reward function specifically for the structural characteristics of small quadrupedal mice, resulting in better control performance. The learning iteration algorithm of this invention adopts an end-to-end control method, which is simpler to implement and performs better when dealing with complex terrain. In project experiments, it was found that reinforcement learning control methods also help improve the movement speed of the robot mouse. The running speed controlled by the learning iteration algorithm on the robot mouse of this invention is approximately 4 times that of the traditional model control method.

[0119] Example 2

[0120] This embodiment provides a motion control system for a small bionic mouse robot based on reinforcement learning, including:

[0121] Minimize effective information extraction module: Based on the structural characteristics of the bionic mouse robot, it constructs an information processing module based on minimizing effective information extraction, and defines the basic elements in the reinforcement learning process, including state S, reward function R, and action space A;

[0122] Decision Execution Module: Used to process the information stream extracted by the Minimize Effective Information Extraction Module and define the network-based control decision-making process;

[0123] The learning iteration module is used to train the robot mouse to acquire a basic walking gait in an initial, simple, barrier-free scenario through a learning iteration process. Finally, based on the training parameters in the barrier-free scenario, the robot mouse is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles.

[0124] In the specific control process, for any time t, the current state is obtained according to the state space definition. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter .

[0125] Specifically, the minimized effective information extraction module includes:

[0126] Sensor data extraction module: used to select the most concise sensor data based on the robotic mouse structure, while meeting performance requirements; speed sensor readings. ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides;

[0127] State space definition module: used to define the state space S = <V,Q, P q >;

[0128] Cumulative Displacement Calculation Module: Used to calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ; Represents the three-dimensional coordinates of the current position;

[0129] The fall penalty function definition module is used to define fall penalty functions. The constant h is a height threshold used to determine whether the robot has fallen.

[0130] The reward function definition module is used to define the reward function based on the cumulative displacement and the fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target;

[0131] Action space definition module: Used to define action space in an end-to-end manner.

[0132]

[0133] Each data component corresponds to the rotation angle of each servo motor of the robotic mouse.

[0134] In addition, the decision execution module includes:

[0135] Initialization module: Used to initialize the Actor network. Through parameters definition ;

[0136] Action space probability distribution acquisition module: used to obtain the current state enter Obtain the probability distribution of the action space. ;

[0137] Action probability selection module: used for probability distribution Sampling to obtain action Select Action probability ;

[0138] Reward value acquisition module: Used by the robot to execute calculated actions and obtain reward values ​​from the environment. .

[0139] The learning iteration module includes:

[0140] Network initialization module: Used to initialize the Actor network in accessible scenarios. parameter and Critic Network parameter ;

[0141] Selection module: Used to obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ;

[0142] Experience buffer pool addition module: used to add Add to experience buffer pool M;

[0143] First iteration module: Used to repeatedly run the selection module and the experience buffer pool addition module until the buffer pool M is full;

[0144] Discount Reward Calculation Module: Used to calculate the discount reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M.

[0145] Training module: Used to extract a mini-batch from buffer pool M for training the network. , ;

[0146] The second iteration module is used to repeatedly run the training module several times, and then clear the experience buffer pool M.

[0147] The third iteration module is used to repeatedly run the selection module, the experience buffer pool addition module, the first iteration module, the discount reward value calculation module, the training module, and the second iteration module several times until training is completed.

[0148] Agent parameter acquisition module: used to obtain parameters based on training. , The above modules are repeatedly run under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

[0149] The training module includes:

[0150] Input module: Used to process data from mini-batch. Input network A new probability distribution is obtained and its cross-entropy ;

[0151] Probability calculation module: used to calculate new probability distributions Calculate probability of occurrence ;

[0152] Evaluation value module: used to... Input network Get evaluation value Then the dominant value function ;

[0153] Loss function calculation module: used to calculate the loss function. ,in , , It is a parameter that indicates importance. Indicates the expected value. Indicates partial derivative, for and The mean square error, ,constant , ;

[0154] Parameter update module: Used to update parameters using gradient descent based on the loss function. parameters and parameters ,in This is the learning rate.

[0155] Example 3

[0156] This embodiment provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the motion control method for a small bionic mouse robot based on reinforcement learning provided in Embodiment 1 above.

[0157] Example 4

[0158] This embodiment provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the motion control method for a small bionic mouse robot based on reinforcement learning provided in Embodiment 1.

[0159] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A motion control method for a small biomimetic mouse robot based on reinforcement learning, characterized in that, Includes the following steps: First, based on the structural characteristics of the bionic mouse robot, an information processing flow based on minimizing effective information extraction was constructed, and the basic elements of the reinforcement learning process were defined, including state S, reward function R, and action space A; based on the information processing of minimizing effective information extraction, a network-based control decision process was defined. Next, in an initial simple, barrier-free scenario, the bionic mouse robot is trained through an iterative learning process to acquire a basic walking gait. Finally, based on the training parameters in the barrier-free scenario, the bionic mouse robot is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles. In the specific control process, for any time t, the current state is obtained according to the definition of the state space. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter ; The information processing flow based on minimizing the extraction of effective information includes: S11. First, based on the bionic mouse robot structure, the most simplified sensing data is selected while meeting performance requirements; speed sensor readings... ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides; Represents the three-dimensional coordinates of the current position; S12. Define the state space S = <V,Q, P q >; S13. Calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ; S14. Define the fall penalty function. The constant h is a height threshold used to determine whether the robot has fallen. S15. Define the reward function based on cumulative displacement and fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target; S16. Define the action space using an end-to-end approach: Each data component corresponds to the rotation angle of each servo motor in the bionic mouse robot.

2. The motion control method for a small bionic mouse robot based on reinforcement learning according to claim 1, characterized in that, The process of decision control execution includes: S21. Initialize the Actor network Through parameters definition ; S22. Change the current state enter Obtain the probability distribution of the action space. ; S23. On probability distribution Sampling to obtain action Select Action probability ; S24. The robot performs the calculated action and receives a reward value from the environment. .

3. The motion control method for a small bionic mouse robot based on reinforcement learning according to claim 2, characterized in that, The learning iteration process specifically includes: S31. First, initialize the Actor network in an accessible scenario. parameter and Critic Network parameter ; S32. Obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ; S33. Will Add to experience buffer pool M; S34. Repeat S32~S33 several times until the buffer pool M is full; S35. Calculate the discounted reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M; S36. Draw a mini-batch from buffer pool M for network training. , ; S37. Repeat step S36 several times, then clear the experience buffer pool M; S38. Repeat steps S32 to S37 several times until training is complete; S39. Based on the parameters obtained after training , Repeat steps S32 to S38 under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

4. The motion control method for a small bionic mouse robot based on reinforcement learning according to claim 3, characterized in that, Step S36 specifically includes: S361. Remove the mini-batch... Input network A new probability distribution is obtained and its cross-entropy ; S362. Using a new probability distribution Calculate probability of occurrence ; S363. [The following appears to be a separate, unrelated sentence:] Will Input network Get evaluation value Then the dominant value function ; S364. Calculate the loss function ,in , , It is a parameter that indicates importance. for and The mean square error, ,constant , ; S365. Update using gradient descent with loss function parameters and parameters ,in This is the learning rate.

5. A motion control system for a small bionic mouse robot based on reinforcement learning, characterized in that, include: Minimize effective information extraction module: Based on the structural characteristics of the bionic mouse robot, it constructs an information processing module based on minimizing effective information extraction, and defines the basic elements in the reinforcement learning process, including state S, reward function R, and action space A; Decision Execution Module: Used to process the information stream extracted by the Minimize Effective Information Extraction Module and define the network-based control decision-making process; The learning iteration module is used to train the bionic mouse robot to acquire a basic walking gait in an initial, simple, barrier-free scenario through a learning iteration process. Finally, based on the training parameters in the barrier-free scenario, the bionic mouse robot is further trained in various obstacle scenarios to acquire an adaptive gait that can overcome obstacles. In the specific control process, for any time t, the current state is obtained according to the state space definition. Through the Actor network Get the current action taken and the selected action probability and reward value In the specific learning iteration process, a number of samples are first taken from the environment using control methods. The data is stored in the empirical buffer M. When M is full, the loss function L is calculated using empirical replay, and then the Actor network is optimized using gradient descent. parameter and Critic Network parameter ; The module for minimizing effective information extraction includes: Sensor data extraction module: used to select the most concise sensor data based on the bionic mouse robot structure, while meeting performance requirements; speed sensor readings. ,in These represent the velocity sensor readings along the x, y, and z axes, respectively; the attitude sensor reading Q, where Q is a quadruple representing the attitude; and the sensor readings P located at each joint of the robot. q It records the hinge rotation angle or the distance the slider slides; Represents the three-dimensional coordinates of the current position; State space definition module: used to define the state space S = <V,Q, P q >; Cumulative Displacement Calculation Module: Used to calculate the cumulative displacement at the current moment by integrating the velocity sensor readings over time. = ; The fall penalty function definition module is used to define fall penalty functions. The constant h is a height threshold used to determine whether the robot has fallen. The reward function definition module is used to define the reward function based on the cumulative displacement and the fall penalty function. Where α and β are parameters representing the importance of each part, This is the unit vector representing the direction of the robot's motion target; Action space definition module: Used to define action space in an end-to-end manner. Each data component corresponds to the rotation angle of each servo motor in the bionic mouse robot.

6. The motion control system for the small bionic mouse robot based on reinforcement learning according to claim 5, characterized in that, The decision execution module includes: Initialization module: Used to initialize the Actor network. Through parameters definition ; Action space probability distribution acquisition module: used to obtain the current state enter Obtain the probability distribution of the action space. ; Action probability selection module: used for probability distribution Sampling to obtain action Select Action probability ; Reward value acquisition module: Used by the robot to execute calculated actions and obtain reward values ​​from the environment. .

7. The motion control system for the small bionic mouse robot based on reinforcement learning according to claim 6, characterized in that, The learning iteration module includes: Network initialization module: Used to initialize the Actor network in accessible scenarios. parameter and Critic Network parameter ; Selection module: Used to obtain the current state through decision control methods. The action of making a choice Select Action probability and reward value ; Experience buffer pool addition module: used to add Add to experience buffer pool M; First iteration module: Used to repeatedly run the selection module and the experience buffer pool addition module until the buffer pool M is full; Discount Reward Calculation Module: Used to calculate the discount reward value for each state in the experience buffer pool M. Where the constant γ is the reward discount factor, This is the final state or the last state added to buffer pool M; Training module: Used to extract a mini-batch from buffer pool M for training the network. , ; The second iteration module is used to repeatedly run the training module several times, and then clear the experience buffer pool M. The third iteration module is used to repeatedly run the selection module, the experience buffer pool addition module, the first iteration module, the discount reward value calculation module, the training module, and the second iteration module several times until training is completed. Agent parameter acquisition module: used to obtain parameters based on training. , The above modules are repeatedly run under different obstacle scenarios to obtain agent parameters that adapt to different scenarios.

8. The motion control system for the small bionic mouse robot based on reinforcement learning according to claim 7, characterized in that, The training module includes: Input module: Used to process data from mini-batch. Input network A new probability distribution is obtained and its cross-entropy ; Probability calculation module: used to calculate new probability distributions Calculate probability of occurrence ; Evaluation value module: used to... Input network Get evaluation value Then the dominant value function ; Loss function calculation module: used to calculate the loss function. ,in , , It is a parameter that indicates importance. Indicates the expected value. Indicates partial derivative; for and The mean square error, ,constant , ; Parameter update module: Used to update parameters using gradient descent based on the loss function. parameters and parameters ,in This is the learning rate.

Images