Robot walking control method, device and equipment and medium

Abstract

The invention discloses a robot walking control method and device, equipment and a medium, and relates to the field of reinforcement learning, and the method comprises the steps: determining a target loss function based on KL divergence constraint, determining an original loss function of a GRPO algorithm by using an L2 parameter regularization method based on real-time environment data and actions of the robot and a numerical feedback signal used for guiding the robot to learn walking; determining an improved loss function according to the target loss function and the original loss function, and adjusting the initial learning rate according to the loss value change to obtain an adjusted learning rate; updating the previous strategy parameter based on the adjusted learning rate and the improved loss function to obtain a current strategy parameter; and when the strategy parameter is updated, determining a strategy gradient based on the original loss function, cutting the strategy gradient, and training the GRPO algorithm through the cut gradient vector and the current strategy parameter so as to control the robot to walk based on the trained GRPO algorithm. And the stability of the algorithm is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning, in particular to a robot walking control method, device, equipment and medium. BACKGROUND

[0002] As a kind of reinforcement learning algorithm, GRPO (Generalized Reinforcement Policy Optimization) aims to maximize cumulative reward by continuously optimizing policy, but there are still several key problems: the stability of policy update is insufficient, there is lack of effective constraint mechanism, which easily leads to training fluctuation or even collapse; Convergence speed is interfered by extreme value and is slow, especially in complex tasks or high-dimensional state space; Generalization ability is limited, and fixed hyperparameters are difficult to adapt to different task characteristics; Gradient explosion increases training instability; Dynamic adjustment mechanism is missing, and static parameter setting cannot flexibly meet training needs, which restricts algorithm performance. These problems jointly affect the training effect, reliability and adaptability of GRPO. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a robot walking control method, device, equipment and medium, which can solve the problems existing in the current GRPO algorithm in the robot walking control process and improve the stability and convergence speed of the algorithm. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a robot walking control method, comprising:

[0005] According to the KL divergence, a target loss function based on the KL divergence constraint is determined, and an L2 parameter regularization method is used to determine an original loss function of the GRPO algorithm based on real-time environmental data of the robot, actions and numerical feedback signals for guiding the robot to learn walking;

[0006] According to the target loss function and the original loss function, an improved loss function is determined, the initial learning rate is dynamically adjusted according to the loss value change in the training process of the GRPO algorithm, and an adjusted learning rate is obtained;

[0007] Based on the adjusted learning rate and the improved loss function, the last policy parameter is updated to obtain the current policy parameter;

[0008] In the process of updating the policy parameter, the policy gradient is determined based on the original loss function, the policy gradient is clipped to obtain a clipped gradient vector, and the GRPO algorithm is trained through the clipped gradient vector and the current policy parameter, so as to control the robot walking based on the corresponding trained GRPO algorithm.

[0009] Optionally, the target loss function based on the KL divergence constraint is determined according to the KL divergence, and includes:

[0010] The KL divergence is determined according to the current policy and the last policy.

[0011] The target loss function based on the KL divergence constraint is determined based on the product of the KL divergence and a preset adaptive coefficient.

[0012] Optionally, the original loss function of the GRPO algorithm is determined based on real-time environment data, actions and numerical feedback signals for guiding the robot to learn to walk by using an L2 parameter regularization method, and includes:

[0013] The original loss function of the GRPO algorithm is determined based on real-time environment data, actions and numerical feedback signals for guiding the robot to learn to walk by using an L2 parameter regularization method through an original loss function determination formula, and the original loss function determination formula is:

[0014] ;

[0015] Wherein, is the original loss function; G is the number of sampled trajectories; is the number of states of the i-th sampled trajectory; is the advantage function; is the clipping parameter; is the policy ratio, wherein is the current policy; is the last policy; is the regularization coefficient; is the L2 regularization term, wherein, is the j-th element of the policy parameter θ.

[0016] Optionally, the improved loss function is determined according to the target loss function and the original loss function, and includes:

[0017] The sum of the target loss function and the original loss function is determined as the improved loss function.

[0018] Optionally, the initial learning rate is dynamically adjusted according to the loss value change in the GRPO algorithm training process to obtain an adjusted learning rate, and includes:

[0019] The initial learning rate is dynamically adjusted according to the loss value change in the GRPO algorithm training process to obtain an adjusted learning rate through an adjusted learning rate determination formula, and the adjusted learning rate determination formula is:

[0020] ;

[0021] wherein, is the adjusted learning rate; is the initial learning rate; t is the current iteration number; T is the total iteration number; is the decay coefficient.

[0022] Optionally, the updating the previous policy parameter based on the adjusted learning rate and the improved loss function to obtain the current policy parameter comprises:

[0023] determining a target product between the improved loss function and the adjusted learning rate;

[0024] determining the current policy parameter based on a sum of the target product and the previous policy parameter.

[0025] Optionally, the clipping the policy gradient to obtain a clipped gradient vector comprises:

[0026] the policy gradient is clipped by a clipped gradient vector determination formula to obtain the clipped gradient vector; the clipped gradient vector determination formula is:

[0027] ;

[0028] wherein, is the clipped gradient vector; is the policy gradient; c is a clipping threshold.

[0029] In a second aspect, the present application discloses a robot walking control device, comprising:

[0030] a loss function determination module configured to determine a target loss function based on KL divergence constraint according to KL divergence, and determine an original loss function of a GRPO algorithm based on real-time environment data of the robot, actions and numerical feedback signals for guiding the robot to learn walking by using an L2 parameter regularization method;

[0031] a learning rate adjustment module configured to determine an improved loss function according to the target loss function and the original loss function, and dynamically adjust an initial learning rate according to loss value changes in a GRPO algorithm training process to obtain an adjusted learning rate;

[0032] a policy parameter updating module configured to update a previous policy parameter based on the adjusted learning rate and the improved loss function to obtain a current policy parameter;

[0033] The algorithm training module is configured to, in the process of updating the policy parameter, determine a policy gradient based on the original loss function, clip the policy gradient to obtain a clipped gradient vector, and train the GRPO algorithm based on the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm.

[0034] In a third aspect, the present application discloses an electronic device, comprising:

[0035] a memory configured to store a computer program;

[0036] a processor configured to execute the computer program to implement the robot walking control method as described above.

[0037] In a fourth aspect, the present application discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the robot walking control method as described above is implemented.

[0038] The present application first determines a target loss function based on KL divergence constraint according to KL divergence, and determines an original loss function of the GRPO algorithm based on real-time environment data of the robot, actions and numerical feedback signals for guiding the robot to learn to walk by using an L2 parameter regularization method. An improved loss function is determined according to the target loss function and the original loss function. The initial learning rate is dynamically adjusted according to the loss value change in the training process of the GRPO algorithm, and an adjusted learning rate is obtained. The last policy parameter is updated based on the adjusted learning rate and the improved loss function to obtain a current policy parameter. In the process of updating the policy parameter, a policy gradient is determined based on the original loss function, the policy gradient is clipped to obtain a clipped gradient vector, and the GRPO algorithm is trained based on the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm. It can be seen that the present application controls the robot walking by improving the GRPO algorithm, which can prevent the size of the update policy amplitude from affecting the robot walking state when the GRPO algorithm updates the policy, and improve the stability of the robot walking. At the same time, the GRPO algorithm is prevented from oscillating in the training process. The improvement of the algorithm reduces the influence of extreme values, so that the algorithm can more stably update towards the optimal policy direction. By adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhance its generalization ability, and finally improve the stability and convergence speed of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0040] Figure 1 A flow chart of a robot walking control method disclosed in the present application;

[0041] Figure 2 A structural schematic diagram of a robot walking control device disclosed in the present application;

[0042] Figure 3 A structural diagram of an electronic device disclosed in the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] At present, when controlling the robot to walk, the existing GRPO algorithm has the problems of insufficient strategy update stability, lack of effective constraint mechanism, slow convergence speed disturbed by extreme value, limited generalization ability, intensified gradient explosion risk, training instability, and lack of dynamic adjustment mechanism when calculating the strategy ratio, thereby causing the robot to suddenly change the walking mode such as gait, step size, etc., leading to falling down and being unable to continue to complete the task, and further unable to obtain subsequent reward signals, which seriously affects the stability of training. In order to solve the above technical problems, the present application provides a robot walking control method, device, equipment and medium, which can solve the problems of the existing GRPO algorithm in the current robot walking control process, improve the stability and convergence speed of the algorithm, and ensure the stability of the robot walking.

[0045] Referring to Figure 1 The embodiments of the present application disclose a robot walking control method, which comprises:

[0046] In step S11, a target loss function based on KL divergence constraint is determined according to the KL divergence, and an L2 parameter regularization method is used to determine an original loss function of the GRPO algorithm based on real-time environment data, actions of the robot and numerical feedback signals for guiding the robot to learn to walk.

[0047] In this embodiment, in each training iteration, a batch of sample data is collected wherein is observation (real-time environment data of the robot), is action, is reward (a numerical feedback signal for guiding the robot to learn to walk). The application introduces an adaptive KL divergence term on the basis of the original GRPO loss function. The KL divergence is calculated as follows: wherein is the last policy, is the current policy. The loss function with adaptive KL divergence constraint is: wherein β is an adaptive coefficient that is dynamically adjusted according to the situation of policy update. Specifically, β can be adjusted by monitoring the amplitude of policy update, for example, when the amplitude of policy update is too large, β is increased to limit the update step. In general, the application determines the KL divergence (Kullback-Leibler Divergence, KLD) according to the current policy and the last policy; and determines the target loss function based on KL divergence constraint based on the product of the KL divergence and a preset adaptive coefficient.

[0048] Then, since the original GRPO loss function can be expressed as:

[0049] ;

[0050] wherein G is the number of sampled trajectories; is the number of states of the i-th sampled trajectory; is the advantage function. However, the application adopts a new normalization method when calculating the policy ratio . Let , the ratio is normalized as follows: . In the loss function, the original policy ratio is replaced by to reduce the influence of extreme values on the algorithm. At the same time, a regularization term is introduced in the policy update to reduce the amplitude of policy update, thereby improving the stability of the policy. Specifically, a regularization term can be added to the policy update formula, so that the original loss function determination formula determines the original loss function of the GRPO algorithm based on the real-time environment data of the robot, the action and the numerical feedback signal for guiding the robot to learn to walk by using the L2 parameter regularization method; the original loss function determination formula is:

[0051] ;

[0052] wherein is the original loss function; G is the number of sampled trajectories; is the number of states of the i-th sampled trajectory; is the advantage function; is the clipping parameter; is a strategy ratio, where is a current strategy; is a last strategy; is a regularization coefficient; is an L2 regularization term, where, is the jth element of the strategy parameter θ.

[0053] In this way, the present application introduces adaptive KL divergence constraint and optimizes the strategy ratio calculation method, aiming to improve the stability and convergence speed of the algorithm, and enhance the generalization ability of the algorithm, so that it can be better applied to various complex reinforcement learning tasks. Specifically, the adaptive KL divergence constraint can prevent the strategy from updating too much and avoid the algorithm from shaking during the training process. At the same time, the optimized strategy ratio calculation method reduces the influence of extreme values, so that the algorithm can more stably update towards the optimal strategy direction; by adaptively adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhancing its generalization ability. Further, it ensures that the robot can stably obtain subsequent reward signals during walking, ensuring the stability of training and the stability of robot walking.

[0054] Step S12, determining an improved loss function according to the target loss function and the original loss function, dynamically adjusting the initial learning rate according to the loss value change in the GRPO algorithm training process to obtain an adjusted learning rate.

[0055] In this embodiment, after determining the target loss function and the original loss function, the improved GRPO loss function is That is, the sum of the target loss function and the original loss function is determined as the improved loss function.

[0056] In addition, in order to improve the adaptability and convergence speed of the algorithm, a mechanism of dynamically adjusting the learning rate is introduced. The learning rate α can be adjusted according to the loss value change in the training process. The adjusted learning rate is determined by the adjusted learning rate determination formula according to the loss value change in the GRPO algorithm training process to obtain the adjusted learning rate; the adjusted learning rate determination formula is:

[0057] ;

[0058] where, is the adjusted learning rate; is the initial learning rate; t is the current iteration number; T is the total iteration number; is a decay coefficient. Thus, the adjusted learning rate is determined.

[0059] Step S13, updating the last policy parameter based on the adjusted learning rate and the improved loss function to obtain a current policy parameter.

[0060] In this embodiment, in order to further optimize the policy updating process, an improved policy updating formula is introduced:

[0061] ;

[0062] wherein, is the adjusted learning rate; is the gradient of the improved loss function with respect to the policy parameter θ; is the current policy parameter; is the last policy parameter.

[0063] Step S14, in the process of updating the policy parameter, determining a policy gradient based on the original loss function, clipping the policy gradient to obtain a clipped gradient vector, and training the GRPO algorithm through the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm.

[0064] In this embodiment, gradient clipping is introduced in the policy updating process to prevent gradient explosion and improve the stability of training. Specifically, after calculating the gradient, the gradient can be clipped:

[0065] ;

[0066] wherein, is the clipped gradient vector; is the policy gradient; and c is a clipping threshold. Gradient clipping can be realized in the following manner: assuming that the gradient vector is then the clipped gradient vector is the i-th element of the clipped gradient vector

[0067] .

[0068] Finally, the current policy is updated, and then the GRPO algorithm is trained by the clipped gradient vector and the current policy parameter, the number of update iterations t = t + 1, if t < T, return to the step of determining the target loss function based on the KL divergence constraint according to the KL divergence, and continue to train; otherwise, the training is completed. Finally, the robot is controlled to walk based on the corresponding trained GRPO algorithm. It needs to be particularly pointed out that the improvement of the GRPO algorithm in the process of controlling the robot to walk in the present application is not only suitable for the application scenario of robot walking, but also suitable for a series of reinforcement learning tasks such as robot control, automatic driving, game AI (Artificial Intelligence, Artificial Intelligence), etc., which can significantly improve the training stability and convergence speed.

[0069] In summary, the present application first determines the target loss function based on the KL divergence constraint according to the KL divergence, determines the original loss function of the GRPO algorithm based on the real-time environmental data of the robot, the action and the numerical feedback signal for guiding the robot to learn to walk by using the L2 parameter regularization method; the improved loss function is determined according to the target loss function and the original loss function, the initial learning rate is dynamically adjusted according to the loss value change in the training process of the GRPO algorithm, and the adjusted learning rate is obtained; the last policy parameter is updated based on the adjusted learning rate and the improved loss function to obtain the current policy parameter; in the process of updating the policy parameter, the policy gradient is determined based on the original loss function, the policy gradient is clipped to obtain the clipped gradient vector, and the GRPO algorithm is trained by the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm. It can be seen that the robot walking is controlled by improving the GRPO algorithm, which can prevent the influence of the size of the update policy amplitude on the robot walking state when the GRPO algorithm updates the policy, and improve the stability of the robot walking. At the same time, the GRPO algorithm is prevented from appearing in the training process. The improvement of the algorithm reduces the influence of the extreme value, so that the algorithm can more stably update towards the optimal policy direction; by adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhance its generalization ability, and finally improve the stability and convergence speed of the algorithm.

[0070] Referring to Figure 2 The robot walking control device disclosed by the embodiment of the present application comprises:

[0071] The loss function determination module 11 is configured to determine a target loss function based on a KL divergence constraint according to the KL divergence, and determine an original loss function of a GRPO algorithm based on real-time environmental data of a robot, an action and a numerical feedback signal for guiding the robot to learn to walk by using an L2 parameter regularization method.

[0072] The learning rate adjustment module 12 is configured to determine an improved loss function according to the target loss function and the original loss function, dynamically adjust an initial learning rate according to a loss value change in a GRPO algorithm training process, and obtain an adjusted learning rate.

[0073] The policy parameter updating module 13 is configured to update a previous policy parameter based on the adjusted learning rate and the improved loss function, to obtain a current policy parameter.

[0074] The algorithm training module 14 is configured to determine a policy gradient based on the original loss function in the process of updating the policy parameter, clip the policy gradient to obtain a clipped gradient vector, and train the GRPO algorithm by using the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm.

[0075] It can be seen that, by improving the GRPO algorithm to control the robot to walk, the robot walking stability can be improved, and the influence of the update policy amplitude on the robot walking state during the update of the GRPO algorithm can be prevented. At the same time, the GRPO algorithm can avoid oscillation during the training process. The improvement of the algorithm reduces the influence of extreme values, so that the algorithm can more stably update towards the optimal policy direction. By adjusting the KL divergence coefficient, the algorithm can better adapt to different environments and tasks, enhance the generalization ability, and finally improve the stability and convergence speed of the algorithm.

[0076] In some specific embodiments, the loss function determination module 11 can be specifically configured to determine a KL divergence according to a current policy and a previous policy, and determine a target loss function based on the KL divergence constraint based on a product of the KL divergence and a preset adaptive coefficient.

[0077] In some specific embodiments, the loss function determination module 11 can be specifically configured to determine an original loss function of the GRPO algorithm based on real-time environment data of the robot, actions, and numerical feedback signals for guiding the robot to learn to walk by using an L2 parameter regularization method through an original loss function determination formula.

[0078] ;

[0079] wherein, is the original loss function; G is the number of sampling trajectories; is the number of states of the i th sampling trajectory; is an advantage function; is a clipping parameter; is a policy ratio, wherein is a current policy; is a previous policy; is a regularization coefficient; is an L2 regularization term, wherein, is the jth element of the policy parameter θ.

[0080] In some specific embodiments, the learning rate adjustment module 12 can be specifically configured to determine the sum of the target loss function and the original loss function as the improved loss function.

[0081] In some specific embodiments, the learning rate adjustment module 12 can be specifically configured to determine the adjusted learning rate by dynamically adjusting the initial learning rate according to the loss value change in the GRPO algorithm training process through an adjusted learning rate determination formula, to obtain the adjusted learning rate; the adjusted learning rate determination formula is:

[0082] ;

[0083] wherein, is the adjusted learning rate; is the initial learning rate; t is the current iteration number; T is the total iteration number; is a decay coefficient.

[0084] In some specific embodiments, the policy parameter updating module 13 can be specifically configured to determine a target product between the improved loss function and the adjusted learning rate; and determine the current policy parameter based on the sum of the target product and the previous policy parameter.

[0085] In some specific embodiments, the algorithm training module 14 can be specifically configured to clip the policy gradient through a clipped gradient vector determination formula to obtain a clipped gradient vector; the clipped gradient vector determination formula is:

[0086] ;

[0087] wherein, is the clipped gradient vector; is the policy gradient; c is a clipping threshold.

[0088] Further, the embodiments of the present application also disclose an electronic device, Figure 3 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0089] Figure 3A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the robot walking control method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiments of the present application can be specifically an electronic computer.

[0090] In the embodiments of the present application, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not specifically limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited here.

[0091] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0092] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the robot walking control method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0093] Further, the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by the processor to implement the robot walking control method disclosed above. For the specific steps of the method, refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0094] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts refer to the method part.

[0095] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. Having thus described the functionality of the examples, a person of ordinary skill in the art will be able to implement such functions in hardware and / or software, using the means and methods available to those skilled in the art. The examples described herein are not meant to limit the scope of the application, but merely to provide examples of the methods and systems being described.

[0096] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0097] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are more especially used for the purpose of distinction from other elements in the specification. Also, the terms "comprise", "include" or "contain" or any other variant thereof are intended to encompass non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise, include, or contain a list of elements are not limited to those elements, but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element defined by the phrase "comprising a... " does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0098] The above has introduced the technical solutions provided by the present application in detail, and the principles and implementation manners of the present application have been described by using specific examples; the above example descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed; in conclusion, the content of the present description should not be understood as limiting the present application.

Claims

1. A robot walking control method characterized by comprising: The method comprises the following steps: determining a target loss function based on KL divergence constraint according to KL divergence, determining an original loss function of GRPO algorithm based on real-time environment data, action and numerical feedback signal for guiding robot to learn walking of the robot by using L2 parameter regularization method; determining an improved loss function according to the target loss function and the original loss function, dynamically adjusting an initial learning rate according to loss value change in the training process of GRPO algorithm to obtain an adjusted learning rate; updating a previous policy parameter based on the adjusted learning rate and the improved loss function to obtain a current policy parameter; in the process of updating the policy parameter, determining a policy gradient based on the original loss function, clipping the policy gradient to obtain a clipped gradient vector, and training the GRPO algorithm through the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on the corresponding trained GRPO algorithm.

2. The robot walking control method according to claim 1, characterized by, The method comprises the following steps: determining a target loss function based on KL divergence constraint according to KL divergence, determining an original loss function of GRPO algorithm based on real-time environment data, action and numerical feedback signal for guiding robot to learn walking of the robot by using L2 parameter regularization method; determining a target loss function based on KL divergence constraint according to KL divergence, determining an original loss function of GRPO algorithm based on real-time environment data, action and numerical feedback signal for guiding robot to learn walking of the robot by using L2 parameter regularization method; 3. The robot walking control method according to claim 1, characterized by, The method comprises the following steps: determining an original loss function of GRPO algorithm based on real-time environment data, action and numerical feedback signal for guiding robot to learn walking of the robot by using L2 parameter regularization method through an original loss function determination formula; the original loss function determination formula is: ； wherein, is the original loss function; G is the number of sampled trajectories; is the number of states of the i-th sampled trajectory; is the advantage function; is the clipping parameter; is the policy ratio, wherein is the current policy; is the previous policy; is the regularization coefficient; is the L2 regularization term, wherein, is the j-th element of the policy parameter θ.

4. The robot walking control method according to claim 1, characterized by, The method comprises the following steps: determining an improved loss function according to the target loss function and the original loss function, dynamically adjusting an initial learning rate according to loss value change in the training process of GRPO algorithm to obtain an adjusted learning rate; 5. The robot walking control method according to claim 1, characterized by, determining an improved loss function according to the target loss function and the original loss function, dynamically adjusting an initial learning rate according to loss value change in the training process of GRPO algorithm to obtain an adjusted learning rate; The method comprises the following steps: ； wherein, is the adjusted learning rate; is the initial learning rate; t is the current iteration number; T is the total iteration number; is the decay coefficient.

6. The robot walking control method according to claim 1, characterized by, determining a target product between the improved loss function and the adjusted learning rate; determining a current policy parameter based on a sum of the target product and the previous policy parameter. The method comprises the following steps:

7. The robot walking control method according to any one of claims 1 to 6, characterized by, determining a target product between the improved loss function and the adjusted learning rate; determining a current policy parameter based on a sum of the target product and the previous policy parameter. ； wherein, is the clipped gradient vector; is the policy gradient; c is a clipping threshold.

8. A robot walking control device characterized by comprising: The method comprises the following steps: The loss function determination module is configured to determine a target loss function based on KL divergence constraint according to the KL divergence, and determine an original loss function of the GRPO algorithm based on real-time environment data of the robot, actions, and numerical feedback signals for guiding the robot to learn walking by using an L2 parameter regularization method. The learning rate adjustment module is configured to determine an improved loss function according to the target loss function and the original loss function, dynamically adjust an initial learning rate according to a loss value change in a training process of the GRPO algorithm, and obtain an adjusted learning rate. The policy parameter update module is configured to update a previous policy parameter based on the adjusted learning rate and the improved loss function, and obtain a current policy parameter. The algorithm training module is configured to determine a policy gradient based on the original loss function in a process of updating the policy parameter, clip the policy gradient to obtain a clipped gradient vector, and train the GRPO algorithm by using the clipped gradient vector and the current policy parameter, so as to control the robot to walk based on a corresponding trained GRPO algorithm.

9. An electronic device, comprising: The robot walking control method comprises the following steps: a memory is configured to store a computer program; a processor is configured to execute the computer program to implement the steps of the robot walking control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on a computer readable storage medium, and the computer program is executed by a processor to implement the steps of the robot walking control method according to any one of claims 1 to 7.

Images