A robot walking control method, device, equipment and medium

Abstract

The application discloses a robot walking control method and device, equipment and medium, relates to the field of reinforcement learning, and comprises the following steps: determining a target loss function based on KL divergence constraint, determining an original loss function of a GRPO algorithm based on real-time environment data, actions and numerical feedback signals for guiding the robot to learn walking of the robot by using an L2 parameter regularization method; determining an improved loss function according to the target loss function and the original loss function, adjusting an initial learning rate according to a loss value change to obtain an adjusted learning rate; updating a previous policy parameter based on the adjusted learning rate and the improved loss function to obtain a current policy parameter; when the policy parameter is updated, determining a policy gradient based on the original loss function, clipping the policy gradient, training the GRPO algorithm through a clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the trained GRPO algorithm. The stability of the algorithm is improved.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning, and in particular to a robot walking control method, apparatus, device, and medium. Background Technology

[0002] GRPO (Generalized Reinforcement Policy Optimization), as a reinforcement learning algorithm, aims to maximize cumulative reward through continuous policy optimization. However, it still suffers from several key problems: insufficient policy update stability and a lack of effective constraint mechanisms, leading to training fluctuations or even crashes; slow convergence speed due to extreme values, especially in complex tasks or high-dimensional state spaces; limited generalization ability, as fixed hyperparameters struggle to adapt to different task characteristics; gradient explosion risk exacerbating training instability; and the lack of dynamic adjustment mechanisms, with static parameter settings unable to flexibly respond to training demands, thus limiting algorithm performance. These issues collectively affect the training effectiveness, reliability, and adaptability of GRPO. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a robot walking control method, device, equipment, and medium that can solve the problems existing in the GRPO algorithm in the current robot walking control process and improve the stability and convergence speed of the algorithm. The specific solution is as follows:

[0004] In a first aspect, this application discloses a robot walking control method, including:

[0005] The target loss function based on KL divergence constraints is determined based on KL divergence. The original loss function of the GRPO algorithm is determined using the L2 parameter regularization method based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk.

[0006] The improved loss function is determined based on the target loss function and the original loss function. The initial learning rate is dynamically adjusted according to the changes in the loss value during the training process of the GRPO algorithm to obtain the adjusted learning rate.

[0007] The previous policy parameters are updated based on the adjusted learning rate and the improved loss function to obtain the current policy parameters;

[0008] During the process of updating the policy parameters, the policy gradient is determined based on the original loss function, and the policy gradient is clipped to obtain the clipped gradient vector. The GRPO algorithm is trained using the clipped gradient vector and the current policy parameters so as to control the robot's movement based on the corresponding trained GRPO algorithm.

[0009] Optionally, determining the target loss function based on KL divergence constraints according to KL divergence includes:

[0010] Determine the KL divergence based on the current strategy and the previous strategy;

[0011] The target loss function based on KL divergence constraints is determined by the product of the KL divergence and the preset adaptive coefficients.

[0012] Optionally, the step of determining the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot in learning to walk using the L2 parameter regularization method includes:

[0013] The original loss function of the GRPO algorithm is determined using the L2 parameter regularization method based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot's learning to walk. The formula for determining the original loss function is as follows:

[0014] ;

[0015] in, The original loss function is denoted by G; G is the number of sampling trajectories. Let i be the number of states for the i-th sampled trajectory; It is the dominant function; These are the trimming parameters; For strategy ratio, where This is the current strategy; This is the previous strategy; The regularization coefficient is used. For L2 regularization terms, ,in, It is the j-th element of the strategy parameter θ.

[0016] Optionally, determining the improved loss function based on the target loss function and the original loss function includes:

[0017] The improved loss function is determined by the sum of the target loss function and the original loss function.

[0018] Optionally, the step of dynamically adjusting the initial learning rate based on the changes in the loss value during the GRPO algorithm training process to obtain the adjusted learning rate includes:

[0019] The adjusted learning rate is obtained by dynamically adjusting the initial learning rate based on the changes in the loss value during the training process of the GRPO algorithm, according to the adjusted learning rate determination formula:

[0020] ;

[0021] in, The adjusted learning rate; The initial learning rate is t; the current iteration number is t; and the total number of iterations is T. This is the attenuation coefficient.

[0022] Optionally, updating the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters includes:

[0023] Determine the target product between the improved loss function and the adjusted learning rate;

[0024] The current policy parameters are determined based on the sum of the target product and the previous policy parameters.

[0025] Optionally, the step of pruning the policy gradient to obtain the pruned gradient vector includes:

[0026] The policy gradient is clipped using the formula for determining the clipped gradient vector, resulting in a clipped gradient vector. The formula for determining the clipped gradient vector is as follows:

[0027] ;

[0028] in, The cropped gradient vector; denoted as the policy gradient; c is the pruning threshold.

[0029] Secondly, this application discloses a robot walking control device, comprising:

[0030] The loss function determination module is used to determine the target loss function based on KL divergence constraints, and to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk using the L2 parameter regularization method.

[0031] The learning rate adjustment module is used to determine the improved loss function based on the target loss function and the original loss function, and to dynamically adjust the initial learning rate according to the changes in the loss value during the training process of the GRPO algorithm, so as to obtain the adjusted learning rate.

[0032] The policy parameter update module is used to update the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters.

[0033] The algorithm training module is used to determine the policy gradient based on the original loss function during the policy parameter update process, prune the policy gradient to obtain the pruned gradient vector, and train the GRPO algorithm using the pruned gradient vector and the current policy parameters so as to control the robot's movement based on the corresponding trained GRPO algorithm.

[0034] Thirdly, this application discloses an electronic device, including:

[0035] Memory, used to store computer programs;

[0036] A processor is used to execute computer programs to implement robot walking control methods as described above.

[0037] Fourthly, this application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned robot walking control method.

[0038] This application first determines the target loss function based on KL divergence constraints, and then uses L2 parameter regularization to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot's learning to walk. An improved loss function is then determined based on the target loss function and the original loss function. The initial learning rate is dynamically adjusted according to the changes in the loss value during the GRPO algorithm training process to obtain the adjusted learning rate. The previous policy parameters are updated based on the adjusted learning rate and the improved loss function to obtain the current policy parameters. During the policy parameter update process, the policy gradient is determined based on the original loss function, and the policy gradient is clipped to obtain a clipped gradient vector. The GRPO algorithm is then trained using the clipped gradient vector and the current policy parameters to control the robot's walking. Therefore, this application, by improving the GRPO algorithm to control robot walking, can prevent the magnitude of the policy update from affecting the robot's walking state during policy updates, thus improving the stability of robot walking. Meanwhile, the improvements avoid oscillations in the GRPO algorithm during training; the algorithm improvements reduce the impact of extreme values, enabling the algorithm to update more stably toward the optimal policy direction; by adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhance its generalization ability, and ultimately improve the stability and convergence speed of the algorithm. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a flowchart of a robot walking control method disclosed in this application;

[0041] Figure 2 This is a schematic diagram of the structure of a robot walking control device disclosed in this application;

[0042] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Currently, existing GRPO algorithms for controlling robot walking suffer from several problems, including insufficient policy update stability during policy ratio calculation, lack of effective constraint mechanisms, slow convergence speed due to extreme value interference, limited generalization ability, increased risk of gradient explosion exacerbating training instability, and lack of dynamic adjustment mechanisms. These issues can lead to sudden changes in the robot's walking pattern, such as gait and stride size, causing falls and preventing the robot from completing the task and obtaining subsequent reward signals, severely impacting training stability. To address these technical problems, this application provides a robot walking control method, device, equipment, and medium that can solve the problems existing in current GRPO algorithms for robot walking control, improve the algorithm's stability and convergence speed, and ensure the stability of robot walking.

[0045] See Figure 1 As shown, an embodiment of the present invention discloses a robot walking control method, including:

[0046] Step S11: Determine the target loss function based on KL divergence constraints according to KL divergence, and use the L2 parameter regularization method to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk.

[0047] In this embodiment, a batch of sample data is collected in each training iteration. ,in It is observation (of the robot's real-time environmental data), It's an action. This is the reward (a numerical feedback signal used to guide the robot in learning to walk). This application introduces an adaptive KL divergence term into the original GRPO loss function. The KL divergence is calculated as follows: ;in It's the previous strategy. This is the current strategy. Therefore, the loss function for adaptive KL divergence constraints is: β is an adaptive coefficient that is dynamically adjusted based on policy updates. Specifically, β can be adjusted by monitoring the magnitude of policy updates; for example, when the policy update magnitude is too large, β is increased to limit the update step size. In summary, this application determines the Kullback-Leibler Divergence (KLD) based on the current policy and the previous policy; and determines the target loss function constrained by the KLD based on the product of the KLD and the preset adaptive coefficient.

[0048] Then, since the original GRPO loss function can be expressed as:

[0049] ;

[0050] Where G is the number of sampling trajectories; Let i be the number of states for the i-th sampled trajectory; It is the advantage function. However, this application has limitations in calculating the strategy ratio. At that time, a new normalization method is adopted. Let... Normalize the ratio: Used in the loss function The original policy ratio is replaced to reduce the impact of extreme values ​​on the algorithm. Simultaneously, a regularization term is introduced into the policy update to reduce the magnitude of the policy update, thereby improving the stability of the policy. Specifically, a regularization term can be added to the policy update formula. Therefore, the original loss function of the GRPO algorithm is determined using the L2 parameter regularization method based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot's learning to walk. The original loss function determination formula is as follows:

[0051] ;

[0052] in, The original loss function is denoted by G; G is the number of sampling trajectories. Let i be the number of states for the i-th sampled trajectory; It is the dominant function; These are the trimming parameters; For strategy ratio, where This is the current strategy; This is the previous strategy; The regularization coefficient is used. For L2 regularization terms, ,in, It is the j-th element of the strategy parameter θ.

[0053] This application aims to improve the stability and convergence speed of the algorithm by introducing adaptive KL divergence constraints and optimized policy ratio calculation methods, while also enhancing its generalization ability, enabling it to be better applied to various complex reinforcement learning tasks. Specifically, adaptive KL divergence constraints prevent excessive policy updates, avoiding oscillations during training. The optimized policy ratio calculation method reduces the impact of extreme values, allowing the algorithm to update more stably towards the optimal policy. By adaptively adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhancing its generalization ability. This ensures that the robot can stably obtain subsequent reward signals during walking, guaranteeing the stability of training and the robot's walking.

[0054] Step S12: Determine the improved loss function based on the target loss function and the original loss function, and dynamically adjust the initial learning rate according to the changes in the loss value during the GRPO algorithm training process to obtain the adjusted learning rate.

[0055] In this embodiment, after determining the target loss function and the original loss function, the improved GRPO loss function is: That is, the sum of the target loss function and the original loss function is determined as the improved loss function.

[0056] Furthermore, to improve the algorithm's adaptability and convergence speed, a mechanism for dynamically adjusting the learning rate is introduced. The learning rate α can be adjusted based on changes in the loss value during training. The adjusted learning rate is determined by dynamically adjusting the initial learning rate according to changes in the loss value during the GRPO algorithm training process using the following formula:

[0057] ;

[0058] in, The adjusted learning rate; The initial learning rate is t; the current iteration number is t; and the total number of iterations is T. This is the decay coefficient. The adjusted learning rate is then determined based on this.

[0059] Step S13: Update the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters.

[0060] In this embodiment, to further optimize the policy update process, an improved policy update formula is introduced:

[0061] ;

[0062] in, The adjusted learning rate; It is the gradient of the improved loss function with respect to the policy parameter θ; The parameters for the current strategy; This is the parameter for the previous strategy.

[0063] Step S14: During the policy parameter update process, the policy gradient is determined based on the original loss function, the policy gradient is clipped to obtain the clipped gradient vector, and the GRPO algorithm is trained using the clipped gradient vector and the current policy parameters so as to control the robot's movement based on the corresponding trained GRPO algorithm.

[0064] In this embodiment, gradient pruning is introduced during the policy update process to prevent gradient explosion and improve training stability. Specifically, the gradient can be pruned after calculation:

[0065] ;

[0066] in, The cropped gradient vector; Let be the gradient of the policy; c is the clipping threshold. Gradient clipping can be implemented as follows: Let the gradient vector be... Then the clipped gradient vector The i-th element is:

[0067] .

[0068] Finally, update the current policy, and then train the GRPO algorithm using the clipped gradient vector and the current policy parameters. Update the iteration count t = t + 1. If t < T, return to the step of determining the objective loss function based on the KL divergence constraint according to the KL divergence and continue training; otherwise, the training ends. Finally, control the robot to walk based on the corresponding trained GRPO algorithm. It should be noted that the improvement of the GRPO algorithm in the process of controlling the robot to walk in this application is not only applicable to the application scenario of robot walking, but also applicable to a series of reinforcement learning tasks such as robot control, autonomous driving, and game AI (Artificial Intelligence), which can significantly improve the training stability and convergence speed.

[0069] In summary, this application first determines the objective loss function based on the KL divergence constraint according to the KL divergence, and uses the L2 parameter regularization method to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals for guiding the robot to learn to walk; determines the improved loss function according to the objective loss function and the original loss function, dynamically adjusts the initial learning rate according to the change of the loss value during the training process of the GRPO algorithm, and obtains the adjusted learning rate; updates the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters; in the process of updating the policy parameters, determines the policy gradient based on the original loss function, clips the policy gradient to obtain the clipped gradient vector, and trains the GRPO algorithm using the clipped gradient vector and the current policy parameters, so as to control the robot to walk based on the corresponding trained GRPO algorithm. It can be seen that this application controls the robot to walk by improving the GRPO algorithm, which can prevent the magnitude of the policy update from affecting the robot's walking state when the GRPO algorithm updates the policy, and improve the stability of the robot's walking. At the same time, it avoids oscillations in the training process of the GRPO algorithm; the improvement of the algorithm reduces the influence of extreme values, enabling the algorithm to update more stably towards the optimal policy direction; by adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhance its generalization ability, and ultimately improve the stability and convergence speed of the algorithm.

[0070] See Figure 2 As shown, an embodiment of the present invention discloses a robot walking control device, including:

[0071] A loss function determination module 11, configured to determine an objective loss function based on the KL divergence constraint according to the KL divergence, and use the L2 parameter regularization method to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals for guiding the robot to learn to walk;

[0072] The learning rate adjustment module 12 is used to determine the improved loss function based on the target loss function and the original loss function, and to dynamically adjust the initial learning rate according to the change of loss value during the training process of the GRPO algorithm, so as to obtain the adjusted learning rate.

[0073] The policy parameter update module 13 is used to update the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters.

[0074] The algorithm training module 14 is used to determine the policy gradient based on the original loss function during the policy parameter update process, prune the policy gradient to obtain the pruned gradient vector, and train the GRPO algorithm using the pruned gradient vector and the current policy parameters so as to control the robot to walk based on the corresponding trained GRPO algorithm.

[0075] As can be seen, this application improves the GRPO algorithm to control robot walking, preventing the magnitude of policy updates from affecting the robot's walking state and improving walking stability. Simultaneously, it avoids oscillations during GRPO training; the algorithm improvements reduce the impact of extreme values, enabling more stable updates towards the optimal policy; and by adjusting the KL divergence coefficient, the algorithm better adapts to different environments and tasks, enhancing its generalization ability and ultimately improving stability and convergence speed.

[0076] In some specific embodiments, the loss function determination module 11 can be used to determine the KL divergence based on the current policy and the previous policy; and to determine the target loss function based on the KL divergence constraint based on the product of the KL divergence and the preset adaptive coefficient.

[0077] In some specific embodiments, the loss function determination module 11 can be used to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk, using the L2 parameter regularization method and the original loss function determination formula; the original loss function determination formula is:

[0078] ;

[0079] in, The original loss function is denoted by G; G is the number of sampling trajectories. Let i be the number of states for the i-th sampled trajectory; It is the dominant function; These are the trimming parameters; For strategy ratio, where This is the current strategy; This is the previous strategy; The regularization coefficient is used. For L2 regularization terms, ,in, It is the j-th element of the strategy parameter θ.

[0080] In some specific embodiments, the learning rate adjustment module 12 can be used to determine the improved loss function by summing the target loss function and the original loss function.

[0081] In some specific embodiments, the learning rate adjustment module 12 can be used to dynamically adjust the initial learning rate based on the changes in the loss value during the GRPO algorithm training process using an adjusted learning rate determination formula, thereby obtaining an adjusted learning rate; the adjusted learning rate determination formula is:

[0082] ;

[0083] in, The adjusted learning rate; The initial learning rate is t; the current iteration number is t; and the total number of iterations is T. This is the attenuation coefficient.

[0084] In some specific embodiments, the policy parameter update module 13 can be used to determine the target product between the improved loss function and the adjusted learning rate; and to determine the current policy parameter based on the sum of the target product and the previous policy parameter.

[0085] In some specific embodiments, the algorithm training module 14 can be used to clip the policy gradient using a clipped gradient vector determination formula to obtain a clipped gradient vector; the clipped gradient vector determination formula is:

[0086] ;

[0087] in, The cropped gradient vector; denoted as the policy gradient; c is the pruning threshold.

[0088] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0089] Figure 3This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the robot walking control method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0091] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0092] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the robot walking control method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0093] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed robot walking control method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0097] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A robot walking control method, characterized in that, include: The target loss function based on KL divergence constraints is determined based on KL divergence. The original loss function of the GRPO algorithm is determined using the L2 parameter regularization method based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk. The improved loss function is determined based on the target loss function and the original loss function. The initial learning rate is dynamically adjusted according to the changes in the loss value during the training process of the GRPO algorithm to obtain the adjusted learning rate. The previous policy parameters are updated based on the adjusted learning rate and the improved loss function to obtain the current policy parameters; During the process of updating the policy parameters, the policy gradient is determined based on the original loss function, the policy gradient is clipped to obtain the clipped gradient vector, and the GRPO algorithm is trained using the clipped gradient vector and the current policy parameters so as to control the robot to walk based on the corresponding trained GRPO algorithm. The step of determining the target loss function based on KL divergence constraints includes: Determine the KL divergence based on the current strategy and the previous strategy; The target loss function based on the KL divergence constraint is determined by the product of the KL divergence and the preset adaptive coefficient. The step of determining the improved loss function based on the target loss function and the original loss function includes: The sum of the target loss function and the original loss function is determined as the improved loss function; The step of updating the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters includes: Determine the target product between the improved loss function and the adjusted learning rate; The current policy parameters are determined based on the sum of the target product and the previous policy parameters; The step of pruning the policy gradient to obtain the pruned gradient vector includes: The policy gradient is clipped using the formula for determining the clipped gradient vector, resulting in a clipped gradient vector. The formula for determining the clipped gradient vector is as follows: ； in, The cropped gradient vector; denoted as the policy gradient; c is the pruning threshold.

2. The robot walking control method according to claim 1, characterized in that, The method of using L2 parameter regularization to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk includes: The original loss function of the GRPO algorithm is determined using the L2 parameter regularization method based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot's learning to walk. The formula for determining the original loss function is as follows: ； in, The original loss function is denoted by G; G is the number of sampling trajectories. Let i be the number of states for the i-th sampled trajectory; It is the dominant function; These are the trimming parameters; For strategy ratio, where This is the current strategy; This is the previous strategy; The regularization coefficient is used. For L2 regularization terms, ,in, It is the j-th element of the strategy parameter θ.

3. The robot walking control method according to claim 1, characterized in that, The step of dynamically adjusting the initial learning rate based on the changes in the loss value during the GRPO algorithm training process to obtain the adjusted learning rate includes: The adjusted learning rate is obtained by dynamically adjusting the initial learning rate based on the changes in the loss value during the training process of the GRPO algorithm, according to the adjusted learning rate determination formula: ； in, The adjusted learning rate; The initial learning rate is t; the current iteration number is t; and the total number of iterations is T. This is the attenuation coefficient.

4. A robot walking control device, characterized in that, include: The loss function determination module is used to determine the target loss function based on KL divergence constraints, and to determine the original loss function of the GRPO algorithm based on the robot's real-time environmental data, actions, and numerical feedback signals used to guide the robot to learn to walk using the L2 parameter regularization method. The learning rate adjustment module is used to determine the improved loss function based on the target loss function and the original loss function, and to dynamically adjust the initial learning rate according to the changes in the loss value during the training process of the GRPO algorithm, so as to obtain the adjusted learning rate. The policy parameter update module is used to update the previous policy parameters based on the adjusted learning rate and the improved loss function to obtain the current policy parameters. The algorithm training module is used to determine the policy gradient based on the original loss function during the policy parameter update process, prune the policy gradient to obtain the pruned gradient vector, and train the GRPO algorithm using the pruned gradient vector and the current policy parameters so as to control the robot to walk based on the corresponding trained GRPO algorithm. The loss function determination module is used to determine the KL divergence based on the current strategy and the previous strategy; and to determine the target loss function based on the product of the KL divergence and the preset adaptive coefficient. The learning rate adjustment module is used to determine the improved loss function by summing the target loss function and the original loss function. The policy parameter update module is used to determine the target product between the improved loss function and the adjusted learning rate; The current policy parameters are determined based on the sum of the target product and the previous policy parameters; The algorithm training module is used to clip the policy gradient using a clipped gradient vector determination formula to obtain a clipped gradient vector; the clipped gradient vector determination formula is: ； in, The cropped gradient vector; denoted as the policy gradient; c is the pruning threshold.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the robot walking control method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the robot walking control method as described in any one of claims 1 to 3.

Images