Robot reinforcement learning control strategy training method and device, equipment and storage medium

Abstract

This invention relates to the field of robot control technology, and discloses a method, apparatus, device, and storage medium for training robot reinforcement learning control strategies. The method includes obtaining the initial training configuration and root directory of the training project for the target robot; determining the training framework adaptation configuration and training metric supervision configuration based on the root directory of the training project, and initiating reinforcement learning training; during training, obtaining training log metrics from the training side and physical evaluation metrics from the evaluation side; during the training cycle in which the physical evaluation metrics are obtained, inputting both and historical parameter tuning records into the analysis agent to jointly determine the configuration increment and update the current training configuration; restarting training using the updated current training configuration; and determining the robot reinforcement learning control strategy when the comprehensive evaluation score and each physical margin metric within the most recently preset stable window reach their corresponding thresholds. This application can reduce manual parameter tuning, improve training efficiency, and enhance the physical reliability of training results.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method, apparatus, device, and storage medium for training robot reinforcement learning control strategies. Background Technology

[0002] In related technologies, reinforcement learning control strategies for legged robots are typically trained in a simulation environment, and the motion capabilities of the control strategy are verified through simulation evaluation or real machine testing after training. During the training process, engineers usually need to read training log curves, observe simulation videos, identify gait anomalies, and manually modify training configurations such as reward weights, course learning parameters, entropy coefficients, motion smoothing parameters, and stability reward parameters.

[0003] However, the reinforcement learning training process for legged robots is characterized by long training cycles, multiple parameter dimensions, strong randomness, and complex physical constraints. Manually reading training curves and manually tuning parameters is not only inefficient but also makes it difficult to reuse tuning experience across different robot models, simulation environments, and training frameworks. Related technologies include automatic hyperparameter tuning for reinforcement learning, robot simulation evaluation, stability criteria, curriculum learning, and large language model-assisted robot task generation. However, these technologies typically only address a single stage of the training or evaluation process and do not integrate training log parsing, physical evaluation, intelligent diagnostics, configuration writing, training restart, historical parameter tuning recording, and convergence adjudication into an automated process for training legged robot control strategies. For example, if the quality of a strategy is judged solely based on reinforcement learning training rewards, it cannot directly reflect physical safety indicators such as foot friction margin, zero torque margin, torque smoothness, and posture stability. If only black-box hyperparameter search is used, it is difficult to explain the reasons for training failures and to accumulate reusable parameter tuning experience. If different robot models share the same terrain parameters, the evaluation results may be distorted due to differences in robot geometry, misjudging pseudo-degradation caused by terrain mismatch as real degradation of the control strategy.

[0004] Therefore, how to automatically acquire feedback from the training and evaluation sides during the training process of robot reinforcement learning control strategies, and automatically modify the training configuration based on physically interpretable diagnostics to improve training efficiency and policy reliability, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for training reinforcement learning control strategies for robots, aiming to solve the technical problems in related technologies where reinforcement learning control strategy training for legged robots relies on manual diagnosis and parameter tuning, lacks physically interpretable feedback during the training process, and is prone to distortion in cross-model evaluation.

[0006] To achieve the above objectives, the present invention provides a method for training a robot reinforcement learning control policy, the method comprising: Obtain the initial training configuration and root directory of the training project for the target robot; Based on the root directory of the training project, determine the training framework adaptation configuration and training metric supervision configuration, and start reinforcement learning training using the initial training configuration; During the reinforcement learning training process, training log metrics are obtained as training-side feedback based on the training metric supervision configuration, and physical evaluation metrics are obtained as evaluation-side feedback based on the simulation evaluation pipeline. During the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent. The analysis agent determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment. Restart the reinforcement learning training using the updated current training configuration until the preset convergence condition is met, and determine the robot reinforcement learning control strategy. The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a robot reinforcement learning control policy training device, the device comprising: The acquisition module is used to obtain the initial training configuration and root directory of the training project for the target robot. The startup module is used to determine the training framework adaptation configuration and training metric supervision configuration based on the root directory of the training project, and to start reinforcement learning training using the initial training configuration. An update module is used to obtain training log metrics as training-side feedback based on the training metric supervision configuration during the reinforcement learning training process, and to obtain physical evaluation metrics as evaluation-side feedback based on the simulation evaluation pipeline; during the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent, and the analysis agent jointly determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment; The restart module is used to restart the reinforcement learning training with the updated current training configuration until the preset convergence condition is met, and to determine the robot reinforcement learning control strategy. The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

[0008] Furthermore, to achieve the above objectives, this application also proposes a robot reinforcement learning control strategy training device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot reinforcement learning control strategy training method described above.

[0009] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the robot reinforcement learning control strategy training method described above.

[0010] One or more technical solutions proposed in this application have at least the following technical effects: Compared to related technologies where legged robot reinforcement learning control strategy training relies on manually reading training curves, observing simulation videos, and manually modifying parameters, this application automatically starts and monitors reinforcement learning training through training framework adaptation configuration and training index supervision configuration. During the training cycle when physical evaluation indicators are obtained, training log indicators, physical evaluation indicators, and historical parameter tuning records are jointly input into the analysis agent to jointly determine the configuration increment. Then, the current training configuration is automatically updated and training is restarted based on the configuration increment. Therefore, the manual parameter tuning process can be reduced, and the training efficiency of robot reinforcement learning control strategy training can be improved.

[0011] Furthermore, this application introduces physical evaluation indicators such as foot friction margin, zero torque point margin, torque smoothness, and posture stability during the training process, so that the parameter tuning process can combine training-side indicators and evaluation-side physical feedback, thereby improving the physical interpretability of the parameter tuning process.

[0012] Furthermore, this application generates a set of terrain parameters specific to the model based on the model's geometric parameters, and performs joint convergence judgment by combining the comprehensive evaluation score threshold and each physical margin threshold within the most recent preset stable window. This can avoid the problem of evaluation results being distorted due to differences in the geometric dimensions of different models, and avoid the situation where a single total score meets the standard but the physical safety margin is insufficient, thereby improving the reliability of the training results of the robot's reinforcement learning control strategy. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the first embodiment of the robot reinforcement learning control strategy training method of this application.

[0016] Figure 2 This is a schematic diagram illustrating the process of generating training framework adaptation configuration and training metric supervision configuration in the robot reinforcement learning control strategy training method of this application.

[0017] Figure 3 This is a schematic diagram of the dual-frequency monitoring and configuration update process in the robot reinforcement learning control strategy training method of this application.

[0018] Figure 4 This is a schematic diagram of the physical evaluation and convergence judgment process in the robot reinforcement learning control strategy training method of this application.

[0019] Figure 5 This is a schematic diagram of the module structure of the robot reinforcement learning control strategy training device according to an embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the hardware operating environment involved in the robot reinforcement learning control strategy training device according to an embodiment of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: Obtain the initial training configuration and training project root directory of the target robot; based on the training project root directory, determine the training framework adaptation configuration and training metric supervision configuration, and start reinforcement learning training using the initial training configuration; during the reinforcement learning training process, obtain training log metrics as training-side feedback and physical evaluation metrics as evaluation-side feedback; during the training cycle in which the physical evaluation metrics are obtained, input the training log metrics, the physical evaluation metrics, and historical parameter tuning records into the analysis agent, which jointly determines the configuration increment and updates the current training configuration; restart the reinforcement learning training using the updated current training configuration, and determine the robot reinforcement learning control strategy when the comprehensive evaluation score and each physical margin metric reach their corresponding thresholds within the most recent preset stable window.

[0025] In related technologies, the training of reinforcement learning control strategies for legged robots usually relies on manual reading of training logs and manual parameter tuning. The training process is difficult to automatically combine physical evaluation results and is also difficult to conduct fair evaluations between different robot models.

[0026] This application, when training a robot's reinforcement learning control strategy, determines the training framework adaptation configuration and training metric supervision configuration based on the training project root directory. During training, it automatically acquires training log metrics and physical evaluation metrics. During the training cycle when physical evaluation metrics are acquired, these two metrics, along with historical parameter tuning records, are input into the analysis agent to jointly determine the configuration increment, thereby automatically updating the training configuration and restarting training. Training convergence is determined when the comprehensive evaluation score and each physical margin metric within the most recent preset stable window reach their respective thresholds. Therefore, it avoids the problems of low efficiency in manual parameter tuning, insufficient physical feedback, and inaccurate cross-model evaluation in related technologies.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a server, workstation, personal computer, robot training platform, etc., or an electronic device capable of realizing the above functions, such as a robot reinforcement learning control strategy training device, etc. The following description uses a robot reinforcement learning control strategy training device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a method for training robot reinforcement learning control strategies, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robot reinforcement learning control strategy training method of this application.

[0029] In this embodiment, the robot reinforcement learning control policy training method includes steps S100 to S400: Step S100: Obtain the initial training configuration and training project root directory of the target robot.

[0030] It should be noted that the robot reinforcement learning control strategy training in this application can be initiated by the user on the training platform or triggered by the automated training system according to the training task plan. The target robot can be a quadruped robot, a bipedal robot, or other legged mobile robots. The training task can be tasks such as speed tracking on flat ground, traversing rugged terrain, step crossing, obstacle crossing, speed command tracking, or posture stabilization control.

[0031] The initial training configuration may include reward weights, course learning parameters, entropy coefficients, motion smoothness coefficients, stability reward weights, domain randomization parameters, etc. The training project root directory is used to locate the training project code, training configuration files, training log directory, and checkpoint file directory. In some optional embodiments, the robot's geometric parameters can also be obtained, which may include parameters such as the target robot's body length, leg length, and hip distance.

[0032] In one feasible implementation, when a user submits a training task on the training platform, they can input the following information: target robot model, training task type, training project root directory, simulation environment name, initial training configuration file path, robot URDF model path, reference trajectory file path, maximum number of rounds, upper limit of single-round training cycle, first preset training cycle, second preset training cycle, and preset convergence threshold. Upon receiving the above information, the training device first checks whether the training project root directory is accessible, whether the initial training configuration file exists, and whether the robot URDF model and reference trajectory file are readable, to ensure that subsequent training tasks can be executed.

[0033] Step S200: Based on the root directory of the training project, determine the training framework adaptation configuration and training metric supervision configuration, and start reinforcement learning training using the initial training configuration.

[0034] It is understandable that different training projects may use different simulation environments and algorithm frameworks. For example, the simulation environment may be Isaac Gym or Isaac Lab, and the algorithm framework may be AMP, WAM, HimLoco, or BeyondMimic. Different frameworks differ in training commands, checkpoint formats, training log directories, training configuration formats, and metric key names.

[0035] In this embodiment, the robot reinforcement learning control strategy training device runs a framework probe based on the root directory of the training project, automatically identifies the aforementioned differences, and generates a training framework adaptation configuration and a training metric supervision configuration. The training framework adaptation configuration is used to instruct subsequent actions such as starting training, reading checkpoints, reading training logs, and writing training configurations; the training metric supervision configuration is used to instruct subsequent actions such as parsing key metrics and their health status in the training logs.

[0036] Specifically, the training device can first call the framework probe script to scan the root directory of the training project. If it identifies that the training project contains Isaac Gym task registration files, runner startup entry points, and Python dataclass configurations, it determines that the simulation environment type is Isaac Gym and sets the configuration writing method to abstract syntax tree patching. If it identifies that the training project contains Isaac Lab runtime entry points and YAML configuration files, it determines that the simulation environment type is IsaacLab and sets the configuration writing method to YAML writing.

[0037] Simultaneously, the training device extracts the metric key names actually written to the training log by scanning the training log write statements in the training code, and maps these metric key names to unified metric aliases. For example, "Train / entropy" can be mapped to "entropy", "Train / discriminator_loss" to "disc_loss", and "Train / reward" to "reward". In this way, the subsequent analysis agent reads the standardized aliases, rather than the original key names that are inconsistent within different frameworks.

[0038] The training logs include, but are not limited to, TensorBoard event files, and may also be other structured log files containing metric key names, corresponding metric values, and training steps.

[0039] In step S300, during the reinforcement learning training process, training log metrics are obtained as training-side feedback based on the training metric supervision configuration, and physical evaluation metrics are obtained as evaluation-side feedback based on the simulation evaluation pipeline. During the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent. The analysis agent determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment.

[0040] It should be noted that training log metrics can include training-side metrics such as reward, entropy, discriminator loss, policy loss, and fragment length. Physical evaluation metrics can include evaluation-side metrics such as joint soft limit margin, linear velocity tracking error, angular velocity tracking error, joint power consumption, posture stability, torque smoothness, foot friction margin, and zero torque point margin.

[0041] In one feasible implementation, training log metrics are acquired at a higher frequency to quickly capture training degradation phenomena such as entropy collapse and discriminator overfitting; physical evaluation metrics are acquired at a lower frequency to determine the actual motion capability and physical safety margin of the current control strategy in the simulation environment.

[0042] For example, the training device can call the training log parsing script every 1000 epochs to read the training log curves from the most recent steps and determine whether the entropy value is below the health threshold, whether the discriminator loss exceeds the healthy range, and whether the reward value has not increased for a long time. If, when training reaches the 3000th epoch, the training log parsing result shows entropy=0.14, while the preset health threshold is 0.15, the training device determines that the entropy_collapse health flag has been triggered and enters the training-side rapid diagnostic process.

[0043] For example, the training device can call the simulation evaluation pipeline every 10,000 epochs to quickly evaluate the current checkpoint. If, at the 10,000th epoch, the simulation evaluation result shows that torque_smoothness=0.52, which is lower than the risk threshold of 0.6, the training device determines that the current strategy has a risk of excessive rate of change of action or torque jitter, and inputs this risk flag into the analysis agent.

[0044] Understandably, configuration increments involve modifying certain parameters in the current training configuration, such as increasing the entropy coefficient, adjusting the action smoothness coefficient, modifying the stability reward weight, and adjusting the tracking tolerance.

[0045] In this embodiment, during the training cycle for acquiring physical evaluation metrics, training log metrics, physical evaluation metrics, and historical parameter tuning records can be jointly input into the analysis agent. The analysis agent then outputs a problem description, root cause diagnosis, error direction exclusion list, confidence level, and configuration increment based on the combined training and evaluation feedback. The training device subsequently adapts the configuration increment to the training configuration file based on the training framework.

[0046] Specifically, when the input is the `entropy_collapse` health flag, the analysis agent can combine historical parameter tuning records with a "good" outcome from the knowledge base to output a configuration increment that increases `entropy_coef` and appropriately relaxes `tracking_sigma`. When the input is the `torque_smoothness_low` risk flag, the analysis agent can combine torque smoothness indicators, velocity tracking error, and attitude stability indicators to output a configuration increment that increases `action_smooth_coef` and `stability_reward_weight`, while recording "do not directly decrease kp to avoid increasing tracking error" in the error direction exclusion list.

[0047] Step S400: Reinforcement learning training is restarted using the updated current training configuration until the preset convergence condition is met, and the robot reinforcement learning control strategy is determined.

[0048] The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

[0049] It should be noted that the preset convergence conditions may include the overall evaluation score reaching the overall evaluation threshold, and each physical margin index reaching its corresponding physical margin threshold. To avoid premature termination of training due to a single accidental achievement of the target, the above conditions may also be required to be met continuously within the most recent preset stability window.

[0050] In this embodiment, the training device restarts training after each configuration update and records the training configuration, configuration increment, training metrics, evaluation metrics, problem description, root cause diagnosis, and parameter tuning results for each round of training. When the preset convergence condition is met, the final control strategy and its corresponding checkpoint file are output.

[0051] To further illustrate the operation process of this application, Table 1 provides a set of exemplary training records. In this example, the comprehensive evaluation threshold is 0.35, the physical margin threshold is 0.90, and the preset stability window is 2 rounds.

[0052] Table 1. Examples of training data In the example shown in Table 1, the comprehensive evaluation score of the third and fourth rounds is not less than 0.35, and the friction margin and ZMP margin are not less than 0.90. Therefore, the training device determines that the preset convergence conditions are met for the last two rounds and outputs the robot reinforcement learning control strategy corresponding to the fourth round.

[0053] In this embodiment, reinforcement learning training is initiated and monitored through training framework adaptation configuration and training metric supervision configuration. During the training cycle when physical evaluation metrics are obtained, the configuration increment is jointly determined based on the two-sided feedback formed by training log metrics and physical evaluation metrics, and then the training configuration is automatically updated and training is restarted. At the same time, the stability window convergence judgment is performed based on the comprehensive evaluation score threshold and each physical margin threshold, which can reduce manual parameter tuning and improve the training efficiency and physical reliability of the robot reinforcement learning control strategy.

[0054] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In step S200, determining the training framework adaptation configuration and training metric supervision configuration based on the root directory of the training project includes steps S210 to S230: Step S210: Based on the root directory of the training project, perform a static abstract syntax tree scan on the training project code to extract the indicator key names written to the training log in the training project code.

[0055] It should be noted that static scanning of the abstract syntax tree is used to parse the code structure of the training project without running the training code. The training device can traverse the Python files in the root directory of the training project, match the call nodes written to the training log, and extract the indicator keys from them.

[0056] For example, the training device can scan files such as logger.py, runner.py, train.py, and algorithm.py in the training project, matching call nodes like writer.add_scalar(“Train / entropy”, value, step). If multiple duplicate indicator key names are found, deduplication is performed to obtain the actual set of indicator key names. Because this step extracts indicator key names directly from the training project code, spelling errors or omissions that may occur when manually configuring indicator names can be avoided.

[0057] It's understandable that the metric keys actually written to the training logs may differ depending on the framework. Extracting the true metric keys through static scanning can prevent subsequent parsing failures caused by incorrect manual configuration of metric names.

[0058] Step S220: Based on the indicator key name, determine the target indicator with diagnostic value, and configure the target indicator with a standardized alias, role classification, health threshold and trigger flag to obtain the training indicator supervision configuration.

[0059] It should be noted that standardized aliases are used to unify the key names of different metrics in different training frameworks into names that can be recognized by the analysis agent. Role classification can include primary, secondary, and debug. Health thresholds are used to determine whether a metric is abnormal, and trigger flags are used to indicate the specific type of abnormality.

[0060] For example, if the entropy value in the training log is lower than a preset lower limit, the entropy_collapse flag can be triggered; if the discriminator loss exceeds a preset healthy range, the disc_overfit or disc_collapse flag can be triggered.

[0061] In one feasible implementation, the training metric supervision configuration may include the following fields: original metric key name, normalization alias, metric role, lower health limit, upper health limit, trend judgment window, and trigger flag. Taking the entropy metric as an example, the original metric key name can be "Train / entropy", the normalization alias can be "entropy", the metric role can be "primary", and the lower health limit can be 0.15. If the current value is below 0.15 and the trend is downward, the `entropy_collapse` flag is triggered. Taking the discriminator loss as an example, the original metric key name can be "Train / discriminator_loss", the normalization alias can be "disc_loss", and the health range can be 0.4 to 0.8. When the value is below 0.4, `disc_collapse` is triggered; when the value is above 0.8, `disc_overfit` is triggered.

[0062] Step S230: Identify the simulation environment type, algorithm framework type, training start command, checkpoint path, training log path, and training configuration file format corresponding to the training project code to obtain the training framework adaptation configuration.

[0063] It's important to note that training framework adaptation configurations are used to mask differences between different training frameworks. For example, IsaacGym training configurations can be in Python dataclass format, while IsaacLab training configurations can be in YAML format. For Python dataclass format, the configuration can be written using an abstract syntax tree patch; for YAML format, the configuration can be updated via YAML writes.

[0064] In one feasible implementation, the training framework adaptation configuration may include: simulation environment type, algorithm framework type, training startup command template, latest checkpoint path matching mode, training log event file path matching mode, training configuration file path, configuration file format type, and configuration writing method. For example, for the AMP / IsaacGym combination, the training startup command template could be "pythontrain.py –task{task_name} –headless", and the checkpoint path matching mode could be "runs / {run_name} / model_ The .pt file can be configured to use Python dataclass AST patching. For the IsaacLab combination, the training startup command template can be "python source / standalone / workflows / train.py –task{task_name}", and the configuration can be written using YAML overriding.

[0065] In this embodiment, by generating training metric supervision configuration and training framework adaptation configuration before training starts, subsequent training monitoring, configuration writing and training restart can all be executed based on standardized configuration, thereby improving the adaptability between different training frameworks.

[0066] Based on the first embodiment of this application, please refer to the third embodiment of this application. Figure 3 In step S300, during the reinforcement learning training process, training log metrics are obtained based on the training metric supervision configuration, and physical evaluation metrics are obtained based on the simulation evaluation pipeline. Based on the training log metrics and the physical evaluation metrics, the configuration increment is determined, including steps S310 to S340: Step S310: At each first preset training cycle, the training log is parsed based on the training indicator supervision configuration to obtain a training log indicator snapshot.

[0067] It should be noted that the first preset training period can be 1000 training periods. The training device can periodically call the training log parsing script, read the training log event file, and parse key indicators according to the training metric supervision configuration.

[0068] Specifically, the training device can obtain the current training cycle number and determine whether it is an integer multiple of the first preset training cycle number. If the current training cycle number is 1000, 2000, 3000, etc., the training log parsing script is invoked. The training log parsing script reads the event file in the training log directory, extracts the most recent data according to the original indicator key names in the training indicator supervision configuration, and calculates the current value, mean, trend, and health status of each target indicator.

[0069] Step S320: If the training log indicator snapshot triggers a health flag, then the first configuration increment is determined based on the training log indicator snapshot and historical parameter tuning records.

[0070] Understandably, health indicators suggest that current training metrics are abnormal. For example, a continuously decreasing entropy value below a health threshold may indicate insufficient policy exploration capability; abnormal discriminator loss may indicate discriminator overfitting or collapse.

[0071] Furthermore, the training device can retrieve similar records with an outcome of "good" from the historical parameter tuning records based on health indicators, and input the retrieval results and the current training log indicator snapshot into the analysis agent, which then outputs the first configuration increment.

[0072] For example, in the 3000th training cycle, a snapshot of the training log metrics shows entropy = 0.14, with a decreasing trend, triggering the entropy_collapse flag. The training device uses entropy_collapse as a search keyword and retrieves a record with a "good" outcome from the historical parameter tuning records. This record shows that "after increasing entropy_coef to 0.02 and relaxing tracking_sigma to 0.3, the entropy value rebounded and the reward resumed growth." Based on the current metrics and this historical record, the analysis agent outputs the first configuration increment: entropy_coef is adjusted from 0.01 to 0.02, and tracking_sigma is adjusted from 0.25 to 0.30.

[0073] Step S330: At each second preset training cycle, the current control strategy is evaluated based on the simulation evaluation pipeline to obtain physical evaluation indicators.

[0074] It should be noted that the second preset training period can be 10,000 training periods. The second preset training period is longer than the first preset training period, and is used to periodically obtain the physical performance of the control policy without significantly increasing the evaluation overhead.

[0075] Physical assessment indicators may include joint soft limit margin, linear velocity tracking error, angular velocity tracking error, joint power consumption, posture stability, torque smoothness, foot friction margin, and zero torque point margin. Each indicator can be expressed as a value between zero and one, and a risk indicator is added when it is below a preset risk threshold.

[0076] In one feasible implementation, at the 10,000th training cycle, the training device obtains the latest checkpoint file path and invokes the fast evaluation mode of the simulation evaluation pipeline. The fast evaluation mode can run the current control strategy under a set of preset terrains and multiple random seeds, obtaining an eight-dimensional physical evaluation index and calculating the overall field. If any index is below 0.6, that index is added to the risk_flags. The evaluation results, along with a snapshot of the most recent training log index, are sent to the analysis agent.

[0077] Step S340: The most recent training log metric snapshot, the physical evaluation metric, and the historical parameter tuning record are input into the analysis agent. The analysis agent determines the second configuration increment based on the training side feedback and the evaluation side feedback. The first preset training period is shorter than the second preset training period.

[0078] Understandably, the second configuration increment is obtained by integrating training-side metrics and evaluation-side metrics. Compared to the first configuration increment, the second configuration increment can be used to more comprehensively address issues exposed in the physical evaluation, such as torque jitter, posture instability, high risk of foot slippage, or insufficient zero-torque margin.

[0079] For example, in the 10,000th training cycle, the evaluation results showed torque_smoothness=0.52, zmp_margin=0.71, and overall=0.67, with torque_smoothness below 0.6. Analyzing the agent's performance, considering the increasing reward trend and normal entropy values ​​in the training logs, it was determined that the current problem was not insufficient exploration, but rather an excessively high rate of action change leading to uneven torque. The agent output a second configuration increment: action_smooth_coef was adjusted from 0.005 to 0.010, and stability_reward_weight was adjusted from 0.30 to 0.50, with "Do not directly reduce kp" recorded in the error direction exclusion list.

[0080] In practical applications, the first configuration increment can be determined by the analysis agent based on training log metric snapshots and historical parameter tuning records. During the training cycle of obtaining physical evaluation metrics, the second configuration increment can be obtained by inputting the most recent training log metric snapshot, physical evaluation metrics, and historical parameter tuning records into the analysis agent. The analysis agent then outputs a problem description, root cause diagnosis, error direction exclusion list, confidence level, and configuration increment based on the combined training and evaluation feedback.

[0081] It should be noted that the analytical agent can be an agent capable of performing natural language analysis and providing structured outputs from training logs, evaluation results, and historical parameter tuning records. The output of the analytical agent can be structured data, including problem, root cause, wrong directions, confidence, and param diff.

[0082] Furthermore, before using the analytical agent, the initial language model can be pre-trained or fine-tuned to enable the analytical agent to perform natural language analysis on training logs, evaluation results, and historical parameter tuning records, and output structured data. Specifically, the training device first constructs a training sample set for the analytical agent. Each sample in the training sample set includes sample input and sample output. The sample input includes a snapshot of training log metrics, physical evaluation metrics, health_flags, risk_flags, and historical parameter tuning records matching the health_flags or risk_flags. The sample output includes a problem description corresponding to the sample input, root cause diagnosis, error direction exclusion list, confidence level, and configuration increment.

[0083] After obtaining the training sample set of the analytical agent, the training device uses a basic language model with natural language understanding capabilities as the initial model and performs supervised fine-tuning on the initial model based on the training sample set. During the supervised fine-tuning process, the sample input is used as the model input, and the sample output conforming to a preset structured field format is used as the target output, enabling the initial model to learn the correspondence between training log anomalies, physical evaluation risks, historical parameter tuning records, and configuration increments.

[0084] After supervised fine-tuning, the training device can further calibrate the output format of the fine-tuned model using structured output validation samples, and use validation samples to check whether the problem, root_cause, wrong_directions, confidence, and param_diff outputs of the fine-tuned model meet preset field requirements and whether they can be configured and written into the parsing tool. If the validation results meet the preset conditions, the fine-tuned model, along with the prompt template, field validator, and knowledge base retrieval module, is encapsulated into the analysis agent. Thus, when invoked, the analysis agent can generate executable structured configuration increments based on input training log metrics, physical evaluation metrics, and historical parameter tuning records.

[0085] Here, problem represents the problem found during the current training or evaluation; root_cause represents the root cause diagnosis; wrong_directions represent the parameter tuning directions that should not be used in this diagnosis; confidence represents the confidence level of the parameter tuning recommendations; and param_diff represents the configuration increment.

[0086] It should be noted that the above training or fine-tuning process does not limit the analytical agent to a specific model training method; in other embodiments, an existing model with the same structured input and output capabilities can also be used, and the analytical agent can be formed by combining prompt templates, field validators and knowledge base retrieval modules.

[0087] In one feasible implementation, the input to the analysis agent can include three parts: the first part is a snapshot of training log metrics, such as the current values ​​and trends of reward, entropy, disc_loss, and episode_length; the second part is physical evaluation metrics, such as dof_limits, lin_vel_err, ang_vel_err, dof_power, orientation_stability, torque_smoothness, friction_margin, and zmp_margin; and the third part is historical parameter tuning records, such as several records that match the current health_flags or risk_flags and have a good outcome. Based on the above input, the analysis agent outputs incremental configuration data containing only structured fields.

[0088] In this embodiment, a dual-frequency asynchronous monitoring mechanism is formed by the first preset training period and the second preset training period, which can simultaneously achieve early detection of training anomalies and periodic evaluation of physical performance with low evaluation overhead.

[0089] Based on the first embodiment of this application, in the fourth embodiment of this application, step S300, the incremental update of the current training configuration based on the configuration, includes steps S350 to S360: Step S350: Based on the training framework adaptation configuration, the configuration is incrementally written into the training configuration file.

[0090] It should be noted that the original training configuration file can be backed up before writing the configuration increment. If the training configuration file is in Python dataclass format, the corresponding fields can be located and modified through the abstract syntax tree patch; if the training configuration file is in YAML format, the corresponding fields can be updated through YAML writing.

[0091] For example, for Python data class configurations, the training device can find the assignment node corresponding to `action_smooth_coef`, replace the original value 0.005 with 0.010, and retain the original file backup path. For YAML configurations, the training device can read the YAML object, write `action_smooth_coef:0.010` at the corresponding level, and then save the modified YAML content back to the configuration file. After each write, the training device outputs a write confirmation result, which may include the field name, old value, new value, backup file path, and a write success flag.

[0092] Step S360: Using the deep merging operator, the current training configuration and the configuration increment are combined into the next round of training configuration.

[0093] Specifically, the synthesis formula is as follows: in, Indicates the first Round training configuration, Indicates the first Round analysis of the configuration increment output by the intelligent agent, Indicates the first Round training configuration, This represents the deep merge operator.

[0094] Understandably, the deep merge operator is used to incrementally write configuration information to the corresponding fields of the current training configuration while retaining unmodified configuration items. This avoids overwriting irrelevant configuration items during each round of parameter tuning, thus improving the security of configuration modifications.

[0095] Furthermore, after completing the configuration write, the training device appends a record to the historical parameter tuning log file. This record may include: record number, timestamp, robot model, task type, trigger type, number of training cycles, problem description, training log metric snapshot, physical evaluation metrics, parameter change dictionary, root cause diagnosis, error direction exclusion list, confidence level, and outcome. Initially, the outcome can be empty; at the next checkpoint, the training device fills the outcome back to good, neutral, or bad based on changes in reward, entropy value, or risk_flags count.

[0096] In this embodiment, by analyzing the configuration increment generated by the intelligent agent and writing the configuration increment into the training configuration file through the training framework, the parameter tuning suggestions can be transformed into executable configuration modification actions.

[0097] Based on the first embodiment of this application, in the fifth embodiment of this application, the physical evaluation index includes foot friction margin. The step of obtaining the physical evaluation index based on the simulation evaluation pipeline includes steps S331 to S332: Step S331: Aggregate the contact points of each foot of the target robot to determine the normal force weight corresponding to each foot.

[0098] It should be noted that during the movement of the target robot, each foot may have multiple contact points. Simply averaging all contact points may not reflect the slippage risk of the weight-bearing foot. Therefore, this embodiment first aggregates the contact points within each foot, and then determines the normal force weight based on the resultant normal force of the foot.

[0099] For example, at a certain evaluation moment, the left foreleg and right hindleg of a quadruped robot are under primary weight-bearing conditions, while the right foreleg and left hindleg bear less weight. If the tangential force at the end of a weight-bearing leg approaches the friction limit, the risk of slippage at that leg will have a greater impact on the overall stability. Therefore, when calculating the friction margin, the training device uses the normal force at the end of the leg as a weight, making the contribution of the weight-bearing leg to the overall friction margin higher.

[0100] Step S332: Determine the foot friction margin based on the normal force weight, the resultant normal force of the foot end, the resultant tangential force of the foot end, and the friction coefficient.

[0101] Specifically, the formula for calculating the foot friction margin is as follows: in, Indicates foot index, Indicates the foot index participating in the normalized summation. Indicates the first The resultant normal force at the foot end, Indicates the first The resultant modulus of the tangential force at the foot end is long. Indicates the first The average value of the friction coefficient at the contact point corresponding to the foot end. Indicates the first The weight of the normal force at the foot only. This indicates the friction margin at the foot.

[0102] Before calculating the foot friction margin, the training device constructs a set of effective feet. This set includes only feet that have a valid contact point, are matched to a corresponding foot geometry, have a resultant normal force greater than zero, and whose ratio of the resultant tangential force magnitude to the foot friction limit is not greater than a preset tangential ratio threshold. The normal force weight is normalized within this set of effective feet. If there is no valid contact point, no matching foot geometry, or the set of effective feet is empty, the normalization calculation is not performed, and the foot friction margin is recorded as 0.

[0103] Understandably, the heavier the load-bearing foot, the greater its impact on the robot's overall stability. By weighting the friction margin of each foot with normal force, the slippage risk of the load-bearing foot can be given a higher weight in the comprehensive evaluation.

[0104] In one example, if the resultant normal force of the left forefoot is 80N, the right forefoot is 40N, the left hindfoot is 35N, and the right hindfoot is 85N, then the normal force weights of the left forefoot and right hindfoot are relatively high. If the resultant tangential force modulus of the right hindfoot is larger, the corresponding decrease in friction margin will significantly reduce the overall foot friction margin. When the training device detects that friction_margin is below the risk threshold, it can input this risk flag into the analysis agent. The analysis agent further combines terrain type, velocity tracking error, and attitude stability to determine whether it is necessary to adjust the stability reward item, motion smoothing item, or terrain course parameters.

[0105] In this embodiment, frictional stability is transformed into a continuous physical evaluation index by using foot friction margin, enabling the training device to perform parameter adjustment and diagnosis based on the risk of foot slippage.

[0106] Based on the first embodiment of this application, in the sixth embodiment of this application, the physical evaluation index includes a zero torque point margin. The step of obtaining the physical evaluation index based on the simulation evaluation pipeline includes steps S333 to S334: Step S333: Determine the coordinates of the zero torque point based on the mass, center of mass position, linear acceleration, inertial tensor, angular acceleration, and angular velocity of each dynamic rigid body of the target robot.

[0107] Specifically, the formula for calculating the coordinates of the zero-moment point is as follows: in, Indicates a dynamic rigid body index. Indicates the first A dynamic rigid body mass Indicates the first The position of the center of mass of a rigid body relative to the center of support. , , They represent exist , , Components in direction, , , They represent the first The acceleration along the center of mass of a rigid body is at... , , Components in direction, The vector of gravitational acceleration is in Components in direction, Indicates the first The inertial tensor of a rigid body in the world coordinate system Indicates the first angular acceleration of a rigid body Indicates the first angular velocity of a rigid body and These represent the zero torque point on the horizontal plane. , Coordinates in the direction.

[0108] Step S334: Determine the zero-moment point margin based on the shortest distance from the zero-moment point coordinates to the boundary of the supporting polygon.

[0109] Specifically, the formula for calculating the zero moment margin is as follows: in, This represents the shortest distance from the zero-moment point to the boundary of the supporting polygon. This indicates the diagonal foot distance in the default standing posture. This indicates the zero torque margin.

[0110] Understandably, the closer the zero-moment point is to the boundary of the supporting polygon, the higher the risk of the target robot tipping over or becoming unstable. By normalizing the shortest distance from the zero-moment point to the boundary of the supporting polygon, a continuous stability index can be obtained for comprehensive evaluation and parameter tuning diagnosis.

[0111] In one example, if the target robot moves on wavy terrain and its roll angle changes significantly, causing the zero-moment point to continuously approach the boundary of the supporting polygon, then `zmp_margin` will decrease. When the training device detects that `zmp_margin` is below a preset threshold, it can input this metric along with `orientation_stability` and `torque_smoothness` into the analysis agent. If `orientation_stability` is also low, the analysis agent can determine that the current policy has insufficient posture stability and output a configuration increment to increase the stability reward weight or adjust the course learning progress speed.

[0112] In this embodiment, dynamic stability is transformed into a continuous physical evaluation index through zero torque point margin, enabling the training device to determine whether the training configuration needs to be adjusted based on the stability margin of the target robot.

[0113] Based on the first embodiment of this application, in the seventh embodiment of this application, referring to... Figure 4 In step S400, restarting the reinforcement learning training using the updated current training configuration until a preset convergence condition is met, and determining the robot's reinforcement learning control strategy, includes steps S410 to S430: Step S410: Generate a body-specific terrain parameter set based on the aircraft model's geometric parameters, and run terrain difficulty search and evaluation on the baseline terrain parameters and the body-specific terrain parameter set respectively.

[0114] Specifically, the formula for generating the terrain parameter set specific to the body shape is as follows: in, Represents the set of geometric parameters of the aircraft model. Represents a deterministic mapping function. This refers to the set of terrain parameters specific to the aircraft model, generated from the set of geometric parameters of the aforementioned aircraft model.

[0115] Understandably, different aircraft models have different body lengths, leg lengths, and hip distances, so the same step depth, wave cycle, or obstacle spacing presents different levels of difficulty for different aircraft models. Generating a set of terrain parameters specific to the aircraft model's geometry can reduce the impact of terrain size mismatch on the assessment results.

[0116] In a feasible deterministic mapping embodiment, the step depth can be set to 0.3 to 0.6 times the leg length, and the obstacle spacing can be set to 1 to 2 times the body length. For example, when the target robot's body length is 0.65m and its leg length is 0.28m, the step depth can be set to 0.084 to 0.168m, and the obstacle spacing can be set to 0.65 to 1.30m. The training device can discretely select values ​​within the above range according to a preset step size to form a body-specific terrain parameter set.

[0117] Furthermore, if the current control strategy performs poorly under baseline terrain parameters but recovers under body-specific terrain parameter sets, then the assessment results can be identified as having a pseudo-degradation; if it performs poorly under both types of terrain, then the control strategy can be identified as having a true degradation.

[0118] In the example above, the hip distance can be 0.21m. If the target robot frequently fails on terrain with small obstacle spacing when using the terrain parameters of the old model, but recovers its success rate under the body-specific terrain parameters, the training device can mark this result as pseudo-degeneration to avoid the analysis agent mistakenly attributing the terrain size mismatch to insufficient control strategy capability.

[0119] For example, for a target robot with a body length of 0.65m, a leg length of 0.28m, and a hip distance of 0.21m, the training device can generate step depths, wave periods, and obstacle spacing that match its body size based on the robot's geometric parameters. If the target robot frequently fails on terrain with small obstacle spacing when using the old model's terrain parameters, but recovers its success rate under the body-specific terrain parameters, the training device can mark this result as pseudo-degeneration, preventing the analysis agent from mistakenly attributing the terrain size mismatch to insufficient control strategy capabilities.

[0120] Step S420: Determine the comprehensive evaluation score based on the terrain quality score and terrain difficulty level.

[0121] Specifically, the formula for calculating the comprehensive evaluation score is as follows: in, This represents the overall evaluation score. Indicates the terrain difficulty level. Indicates the terrain difficulty level The average mass score on the surface. Indicates evaluation of random seed, This indicates that the 50th percentile value is taken for multiple sub-evaluation results. This indicates that the average value is taken for the results of different terrain difficulty levels.

[0122] It should be noted that the terrain difficulty level can range from 1 to 10. The training device can perform difficulty searches for different terrain types separately and run multiple random seeds at each difficulty level to reduce the impact of randomness in a single evaluation.

[0123] Step S430: If the comprehensive evaluation score is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold within the most recent preset stable window, then the preset convergence condition is satisfied.

[0124] Specifically, the preset convergence conditions include: in, Indicates the first Round comprehensive evaluation score, This indicates the comprehensive evaluation threshold. Indicates the first Round The evaluation value of each physical margin indicator, Indicates the first The physical margin threshold for each physical margin indicator. This represents the physical margin index.

[0125] It should be noted that the preset stability window can be the evaluation results of several consecutive rounds of training. Training convergence is determined only when the above conditions are met within the most recent preset stability window.

[0126] In this embodiment, by using both the comprehensive evaluation score and the physical margin index as convergence conditions, the situation where the control strategy only meets the target in terms of total score but lacks a certain physical safety margin can be avoided, thereby improving the safety of the final control strategy.

[0127] This application also provides a robot reinforcement learning control strategy training device, please refer to... Figure 5 The robot reinforcement learning control strategy training device includes: Module 10 is used to obtain the initial training configuration and training project root directory of the target robot; The startup module 20 is used to determine the training framework adaptation configuration and training metric supervision configuration based on the root directory of the training project, and to start reinforcement learning training using the initial training configuration. The update module 30 is used to obtain training log metrics as training-side feedback based on the training metric supervision configuration during the reinforcement learning training process, and to obtain physical evaluation metrics as evaluation-side feedback based on the simulation evaluation pipeline; during the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent, and the analysis agent jointly determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment; Restart module 40 is used to restart the reinforcement learning training with the updated current training configuration until the preset convergence condition is met, and to determine the robot reinforcement learning control strategy. The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

[0128] The robot reinforcement learning control strategy training device provided in this application, employing the robot reinforcement learning control strategy training method in the above embodiments, can solve the technical problem of robot reinforcement learning control strategy training. Compared with the prior art, the beneficial effects of the robot reinforcement learning control strategy training device provided in this application are the same as the beneficial effects of the robot reinforcement learning control strategy training method provided in the above embodiments, and other technical features in the robot reinforcement learning control strategy training device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0129] Furthermore, the present invention also proposes a robot reinforcement learning control policy training device, which includes: a memory, a processor, and a robot reinforcement learning control policy training program stored in the memory and executable on the processor. When the robot reinforcement learning control policy training program is executed by the processor, it implements the steps of the robot reinforcement learning control policy training method as described above.

[0130] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing a robot reinforcement learning control policy training device according to embodiments of this application. The robot reinforcement learning control policy training device in these embodiments may include, but is not limited to, servers, workstations, desktop computers, training platforms, robot control development equipment, etc.

[0131] like Figure 6As shown, the robot reinforcement learning control policy training device may include a processing unit 1001, such as a central processing unit or a graphics processing unit, which can execute various appropriate actions and processes according to the program stored in the read-only memory 1002 or the program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the robot reinforcement learning control policy training device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus 1005.

[0132] Typically, the following devices can be connected to the input / output interface 1006: an input device 1007 including, for example, a keyboard, mouse, sensor interface, etc.; an output device 1008 including, for example, a display, speaker, etc.; a storage device 1003 including, for example, a disk, solid-state drive, etc.; and a communication device 1009. The communication device 1009 allows the robot reinforcement learning control policy training device to communicate wirelessly or wiredly with other devices to exchange data.

[0133] Although the figure shows a robot reinforcement learning control policy training device with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented alternatively.

[0134] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0135] The robot reinforcement learning control strategy training device provided in this application, employing the robot reinforcement learning control strategy training method in the above embodiments, can solve the technical problem of robot reinforcement learning control strategy training. Compared with the prior art, the beneficial effects of the robot reinforcement learning control strategy training device provided in this application are the same as the beneficial effects of the robot reinforcement learning control strategy training method provided in the above embodiments, and other technical features in this robot reinforcement learning control strategy training device are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0136] Furthermore, this application also proposes a storage medium, which is a computer-readable storage medium storing computer-readable program instructions for executing the above-described robot reinforcement learning control policy training method, thereby solving the technical problem of robot reinforcement learning control policy training. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the robot reinforcement learning control policy training method provided in the above embodiments, and will not be repeated here.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory or flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in the robot reinforcement learning control policy training device, or it may exist independently and not be assembled into the robot reinforcement learning control policy training device.

[0139] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for training a robot's reinforcement learning control strategy, characterized in that, Includes the following steps: Obtain the initial training configuration and root directory of the training project for the target robot; Based on the root directory of the training project, determine the training framework adaptation configuration and training metric supervision configuration, and start reinforcement learning training using the initial training configuration; During the reinforcement learning training process, training log metrics are obtained as training-side feedback based on the training metric supervision configuration, and physical evaluation metrics are obtained as evaluation-side feedback based on the simulation evaluation pipeline. During the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent. The analysis agent determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment. Restart the reinforcement learning training using the updated current training configuration until the preset convergence condition is met, and determine the robot reinforcement learning control strategy. The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

2. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, Based on the root directory of the training project, the steps for determining the training framework adaptation configuration and training metric supervision configuration include: Based on the root directory of the training project, perform a static scan of the abstract syntax tree on the training project code to extract the indicator key names written to the training log in the training project code. Based on the indicator key names, target indicators with diagnostic value are determined, and standardized aliases, role classifications, health thresholds, and trigger flags are configured for the target indicators to obtain the training indicator supervision configuration. Identify the simulation environment type, algorithm framework type, training start command, checkpoint path, training log path, and training configuration file format corresponding to the training project code to obtain the training framework adaptation configuration.

3. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, During the reinforcement learning training process, training log metrics are obtained based on the supervised configuration of the training metrics, and physical evaluation metrics are obtained based on the simulation evaluation pipeline. The step of determining the configuration increment based on the training log metrics and the physical evaluation metrics includes: At each first preset training cycle, the training log is parsed based on the training metric supervision configuration to obtain a training log metric snapshot; If the training log metric snapshot triggers a health flag, then the first configuration increment is determined based on the training log metric snapshot and historical parameter tuning records; At each second preset training cycle, the current control strategy is evaluated based on the simulation evaluation pipeline to obtain physical evaluation indicators. The most recent training log snapshot, the physical evaluation metric, and the historical parameter tuning record are input into the analysis agent, which jointly determines the second configuration increment, wherein the first preset training period is shorter than the second preset training period.

4. The robot reinforcement learning control strategy training method as described in claim 3, characterized in that, The first preset training period is 1000 training periods, and the second preset training period is 10000 training periods.

5. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, The steps for incrementally updating the current training configuration based on the aforementioned configuration include: Based on the training framework adaptation configuration, the configuration is incrementally written into the training configuration file; Using the deep merging operator, the current training configuration and the configuration increment are combined into the next round of training configuration, wherein the configuration combination formula is: in, Indicates the first Round training configuration, Indicates the first Round analysis of the configuration increment output by the intelligent agent, Indicates the first Round training configuration, This represents the deep merge operator.

6. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, The physical evaluation index includes foot friction margin, and the steps for obtaining the physical evaluation index based on the simulation evaluation pipeline include: Aggregate the contact points of each foot of the target robot to determine the normal force weight corresponding to each foot. Based on the normal force weight, the resultant normal force at the foot end, the resultant tangential force modulus at the foot end, and the friction coefficient, the foot end friction margin is determined, wherein the formula for calculating the foot end friction margin is: in, Indicates foot index, Indicates the foot index participating in the normalized summation. Indicates the first The resultant normal force at the foot end, Indicates the first The resultant modulus of the tangential force at the foot end is long. Indicates the first The average value of the friction coefficient at the contact point corresponding to the foot end. Indicates the first The weight of the normal force at the foot only. Indicates the friction margin at the foot end; The foot end participating in the normalization summation is configured to have an effective contact point, be matched with the corresponding foot end geometry, have a resultant normal force of the foot end greater than zero, and have a ratio of the resultant tangential force of the foot end to the foot end friction limit not greater than a preset tangential ratio threshold. If there is no effective contact point, no matching foot end geometry, or the set of foot ends participating in the normalization summation is empty, then the foot end friction margin is recorded as 0.

7. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, The physical evaluation index includes the zero moment point margin, and the steps for obtaining the physical evaluation index based on the simulation evaluation pipeline include: Based on the mass, center of mass position, linear acceleration, inertia tensor, angular acceleration, and angular velocity of each dynamic rigid body of the target robot, determine the coordinates of the zero torque point; Based on the shortest distance from the zero-moment point coordinates to the boundary of the supporting polygon, the zero-moment point margin is determined, wherein the formulas for calculating the zero-moment point coordinates and the zero-moment point margin are as follows: in, Indicates a dynamic rigid body index. Indicates the first A dynamic rigid body mass Indicates the first The position of the center of mass of a rigid body relative to the center of support. , , They represent exist , , Components in direction, , , They represent the first The acceleration along the center of mass of a rigid body is at... , , Components in direction, The vector of gravitational acceleration is in Components in direction, Indicates the first The inertial tensor of a rigid body in the world coordinate system Indicates the first angular acceleration of a rigid body Indicates the first angular velocity of a rigid body and These represent the zero torque point on the horizontal plane. , Coordinates in direction This represents the shortest distance from the zero-moment point to the boundary of the supporting polygon. This indicates the diagonal foot distance in the default standing posture. This indicates the zero torque margin.

8. The robot reinforcement learning control strategy training method as described in claim 6, characterized in that, The physical evaluation index also includes zero moment point margin, and the step of obtaining the physical evaluation index based on the simulation evaluation pipeline further includes: The coordinates of the zero torque point are determined based on the mass, center of mass position, linear acceleration, inertia tensor, angular acceleration, and angular velocity of each dynamic rigid body of the target robot. The zero-moment point margin is determined based on the shortest distance from the zero-moment point coordinates to the boundary of the supporting polygon, wherein the formulas for calculating the zero-moment point coordinates and the zero-moment point margin are as follows: in, Indicates a dynamic rigid body index. Indicates the first A dynamic rigid body mass Indicates the first The position of the center of mass of a rigid body relative to the center of support. , , They represent exist , , Components in direction, , , They represent the first The acceleration along the center of mass of a rigid body is at... , , Components in direction, The vector of gravitational acceleration is in Components in direction, Indicates the first The inertial tensor of a rigid body in the world coordinate system Indicates the first angular acceleration of a rigid body Indicates the first angular velocity of a rigid body and These represent the zero torque point on the horizontal plane. , Coordinates in direction This represents the shortest distance from the zero-moment point to the boundary of the supporting polygon. This indicates the diagonal foot distance in the default standing posture. Indicates zero torque margin; The foot friction margin and the zero torque point margin are used together as the evaluation side feedback input to the analysis agent to jointly determine the configuration increment.

9. The robot reinforcement learning control strategy training method as described in claim 1, characterized in that, Restarting the reinforcement learning training using the updated current training configuration until a preset convergence condition is met, and determining the robot's reinforcement learning control strategy includes the following steps: Obtain the target robot's model geometric parameters, generate a body-specific terrain parameter set based on the model geometric parameters, and perform terrain difficulty search and evaluation on both the baseline terrain parameters and the body-specific terrain parameter set. The expression for the body-specific terrain parameter set is: in, Represents the set of geometric parameters of the aircraft model. Represents a deterministic mapping function. This represents the set of terrain parameters specific to the aircraft model, generated from the set of geometric parameters of that model. Based on the terrain quality score and terrain difficulty level, the comprehensive evaluation score is determined, expressed as: in, This represents the overall evaluation score. Indicates the terrain difficulty level. Indicates the terrain difficulty level The average mass score on the surface. Indicates evaluation of random seed, This indicates that the 50th percentile value is taken for multiple sub-evaluation results. This indicates that the average value is taken for the results of different terrain difficulty levels; If, within the most recent preset stable window, the overall evaluation score is not lower than the overall evaluation threshold, and each physical margin index is not lower than its corresponding physical margin threshold, then the preset convergence condition is satisfied, wherein: in, Indicates the first Round comprehensive evaluation score, This indicates the comprehensive evaluation threshold. Indicates the first Round The evaluation value of each physical margin indicator, Indicates the first The physical margin threshold for each physical margin indicator. This represents the physical margin index.

10. A robot reinforcement learning control strategy training device, characterized in that, include: The acquisition module is used to obtain the initial training configuration and root directory of the training project for the target robot. The startup module is used to determine the training framework adaptation configuration and training metric supervision configuration based on the root directory of the training project, and to start reinforcement learning training using the initial training configuration. An update module is used to obtain training log metrics as training-side feedback based on the training metric supervision configuration during the reinforcement learning training process, and to obtain physical evaluation metrics as evaluation-side feedback based on the simulation evaluation pipeline; during the training cycle in which the physical evaluation metrics are obtained, the training log metrics, the physical evaluation metrics, and historical parameter tuning records are jointly input into the analysis agent, and the analysis agent jointly determines the configuration increment based on the training-side feedback and the evaluation-side feedback, and updates the current training configuration based on the configuration increment; The restart module is used to restart the reinforcement learning training with the updated current training configuration until the preset convergence condition is met, and to determine the robot reinforcement learning control strategy. The preset convergence condition is configured as follows: the comprehensive evaluation score within the most recent preset stable window is not lower than the comprehensive evaluation threshold and each physical margin index is not lower than the corresponding physical margin threshold.

11. A robot reinforcement learning control strategy training device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot reinforcement learning control policy training method as described in any one of claims 1 to 9.

12. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the robot reinforcement learning control strategy training method as described in any one of claims 1 to 9.

Images