Transition control methods and systems to assist robots in switching control strategies

Abstract

This invention belongs to the technical field of robot transition control. To address the problem of frequent failures when directly switching control strategies in existing robots, it proposes a transition control method and system to assist robots in switching control strategies. Based on a dual-branch predictive encoder using a variational autoencoder, it predicts the robot's future state and changes in the surrounding environment's height. Based on the prediction results, a deep reinforcement learning model is used to obtain the robot's current transition control strategy. A reward evaluation network determines the expected value of the cumulative reward under the current transition control strategy, and a constraint evaluation network determines the expected value of the cumulative constraint violation degree under the current transition control strategy. The strategy network and the dual evaluation network are then trained based on the expected values ​​of the cumulative reward and the cumulative constraint violation degree. This invention improves the reliability of robots in completing complex multi-stage tasks.

Description

Technical Field

[0001] This invention belongs to the technical field of robot transition control, and particularly relates to a transition control method and system for assisting robots in switching control strategies. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Quadruped robots, with their superior adaptability to complex terrain, have demonstrated enormous application potential in scenarios such as disaster relief and industrial inspection. In recent years, control strategies based on deep reinforcement learning have become an important technology for improving the robustness of robot motion. Among them, the PPO algorithm, as a mainstream optimization method, enables robots to stably complete basic motion skills through the trained control strategies.

[0004] In reinforcement learning-driven robot motion, multiple objectives (such as trajectory accuracy, smooth motion, energy efficiency, and obstacle avoidance) often constrain each other. A single evaluation network weights and merges multiple objectives into a single reward, failing to simultaneously address the optimality of different objectives. The reward scales for different objectives vary significantly, easily allowing the dominant objective to mask secondary objectives; furthermore, the optimization gradient directions often contradict each other, and a single value function can only provide a compromise gradient, unable to achieve multi-objective collaborative optimization. Moreover, robots cannot directly access privileged information such as terrain and velocity in real-world environments. If such information is needed to enhance policy decision-making capabilities, estimation must rely on ontological perception data. Traditional methods use deterministic encoding to predict future environmental and robot state changes, which is susceptible to noise interference and has poor generalization ability for observation combinations not present in the training data, thus affecting the normal decision-making of subsequent control strategies. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a transition control method and system to assist robots in switching control strategies, which effectively alleviates the problem of robots being prone to failure when directly switching control strategies at uncertain states, and improves the reliability of robots in completing complex multi-stage tasks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a transition control method for assisting a robot in switching control strategies, comprising: Using the current observation matrix after the robot task switch, a bi-branch predictive encoder based on variational autoencoder is used to predict the robot's future state and the height change of the surrounding environment, respectively. The robot's future state and the height changes of its surrounding environment are predicted using a deep reinforcement learning model to obtain the robot's current transition control strategy, which is then used to control the robot accordingly. The deep reinforcement learning model includes a policy network and a dual evaluation network. The expected value of the cumulative reward under the current transition control policy is determined based on the reward evaluation network, and the expected value of the cumulative constraint violation degree under the current transition control policy is determined based on the constraint evaluation network. The current policy network and the current dual evaluation network are trained based on the expected value of the cumulative reward and the expected value of the cumulative constraint violation degree until the parameters of the policy network and the dual evaluation network both meet the preset stability conditions. The corresponding current transition control policy of the robot is determined based on the latest current policy network.

[0007] In a second aspect, the present invention provides a transition control system for assisting a robot in switching control strategies, comprising: The prediction module is configured to: use the current observation matrix after the robot task switch, and predict the robot's future state and the height change of the surrounding environment through a dual-branch predictive encoder based on a variational autoencoder. The transition control module is configured to: use a deep reinforcement learning model to obtain the robot's current transition control strategy based on the predicted future state of the robot and the height changes of the surrounding environment, and then use the current transition control strategy to control the robot accordingly; wherein, the deep reinforcement learning model includes a policy network and a dual evaluation network, determines the expected value of the cumulative reward under the current transition control strategy based on the reward evaluation network, determines the expected value of the cumulative constraint violation degree under the current transition control strategy based on the constraint evaluation network, and trains the current policy network and the current dual evaluation network based on the expected value of the cumulative reward and the expected value of the cumulative constraint violation degree until the parameters of the policy network and the dual evaluation network both meet the preset stability conditions, and determines the corresponding current transition control strategy for the robot based on the latest current policy network.

[0008] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0009] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0010] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0011] The above one or more technical solutions have the following beneficial effects: In this invention, a dual-branch predictive encoder based on a variational autoencoder synchronously and probabilistically predicts the robot's future trajectory and changes in the surrounding environment's height. This enables the transition strategy to guide the robot to adjust its posture in advance to adapt to environmental changes. A dual evaluation network is designed. The expected value of the cumulative reward under the current transition control strategy is determined by the reward evaluation network, and the expected value of the cumulative constraint violation degree under the current transition control strategy is determined by the constraint evaluation network. The current strategy network and the current dual evaluation network are trained based on the expected values ​​of the cumulative reward and the cumulative constraint violation degree, achieving a specialized evaluation of rewards and constraints. This avoids conflicts in multi-objective optimization caused by a single evaluation network. At the same time, while maximizing rewards to complete the expected task, the strategy also strictly adheres to the constraints of the constraint evaluation network, limiting the occurrence of dangerous actions.

[0012] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0014] Figure 1 This is a flowchart of a transition control method for assisting a robot in switching control strategies, as described in an embodiment of the present invention. Figure 2 This is an overall framework diagram of the transition control method for switching control strategies of the robot in this embodiment of the invention; Figure 3 This is a schematic diagram of the transition of the quadruped robot's execution phase in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the switching between different control strategies in an embodiment of the present invention. Detailed Implementation

[0015] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0016] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0017] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0018] Example 1 In real-world scenarios, robots typically perform multi-stage, complex tasks, requiring dynamic switching between different control strategies. The optimal initial state distributions for different strategies differ significantly. Especially when the preceding environment is unknown and the robot's state is random at the time of strategy switching, directly switching strategies can easily lead to state mismatch issues, causing subsequent task execution to fail. Existing methods often achieve task transitions by constraining the robot to reach a preset termination state. These methods rely on clear stage divisions and state transition conditions between skills. For scenarios with ambiguous transition goals, significant differences in state and goal from preceding tasks, and a lack of explicit switching constraints, existing methods struggle to generate reasonable intermediate transition behaviors and cannot guarantee robust connection of task sequences.

[0019] Based on this, this embodiment discloses a transition control method for assisting a robot in switching control strategies, including: Using the current observation matrix after the robot task switch, a bi-branch predictive encoder based on variational autoencoder is used to predict the robot's future state and the height change of the surrounding environment, respectively. The robot's future state and the height changes of its surrounding environment are predicted using a deep reinforcement learning model to obtain the robot's current transition control strategy, which is then used to control the robot accordingly. The deep reinforcement learning model includes a policy network and a dual evaluation network. The expected value of the cumulative reward under the current transition control policy is determined based on the reward evaluation network, and the expected value of the cumulative constraint violation degree under the current transition control policy is determined based on the constraint evaluation network. The current policy network and the current dual evaluation network are trained based on the expected value of the cumulative reward and the expected value of the cumulative constraint violation degree until the parameters of the policy network and the dual evaluation network both meet the preset stability conditions. The corresponding current transition control policy of the robot is determined based on the latest current policy network.

[0020] In this embodiment, the robot's future trajectory and changes in the surrounding environment's height are predicted simultaneously, enabling the transition strategy to guide the robot to adjust its posture in advance according to the changing environmental trends. A dual evaluation network is designed, which determines the expected value of the cumulative reward under the current transition control strategy based on the reward evaluation network and the expected value of the cumulative constraint violation degree under the current transition control strategy based on the constraint evaluation network. The current strategy network and the current dual evaluation network are trained based on the expected values ​​of the cumulative reward and the cumulative constraint violation degree, achieving specialized evaluation of rewards and constraints. This avoids conflicts in multi-objective optimization caused by a single evaluator, and also ensures that the strategy maximizes rewards to complete the expected task while strictly adhering to the constraints of the constraint evaluation network, limiting the occurrence of dangerous actions.

[0021] The following is combined Figures 2-4The transition control method for assisting the robot in switching control strategies, as proposed in this embodiment, will be described in detail below: Step 1: Based on the robot's state trajectory after switching target control strategies, use the trained discriminator to obtain the task success rate of executing subsequent tasks under the current state trajectory, and add the task success rate to the cumulative reward calculation of the reward evaluation network.

[0022] In this embodiment, an initial control strategy is run until the robot contacts the high platform. Then, the target control strategy is switched directly. The sampled state and task execution data are used as a dataset to train a discriminator, which can score the robot's performance in subsequent tasks based on its initial state.

[0023] Specifically, firstly, in a simulation environment that includes various common terrain conditions and whose endpoint is a high platform, a pre-trained initial control strategy is run. The robot is controlled to move through environments including gentle slopes and steps until it makes contact with a target platform. At the moment of contact, the robot's complete state is recorded, including 3D velocity commands. 3D robot angular velocity 3D robot gravity vector The position of each joint of a 12-dimensional robot 12-dimensional joint velocity Action in the 12th dimension at the previous moment 3D robot actual speed External forces acting on a 3D robot The terrain surrounding the 187-dimensional robot A total of 238 dimensions of data were collected. A basic initial state set was constructed by collecting state samples from N robot contact points at the high platform. .

[0024] for Each initial state in Directly switch the control strategy to the pre-trained target control strategy. Target control strategy It was specifically trained to perform jumps onto high platforms. During each execution, the time from the switch point was recorded. The robot's state trajectory within each of the following 40 time steps. And the binary success label of this attempt The criteria for successful task execution are defined as follows: after 10 seconds, all the robot's foot coordinates are above the high platform, and the distance between the coordinates of the two hind feet and the edge of the platform is greater than 5cm. Each initial state is repeated 20 times, resulting in 20 sets of data. Trajectories with the same initial state are merged and used to calculate the success rate. Where K represents the total number of trials under the same initial conditions, k represents the sequence number of a particular trial, y is 1 for a successful trial and 0 for an unsuccessful trial, therefore The success rate is the number of successful trials divided by K. Finally, a dataset is constructed. That is, each data sample contains a state trajectory. and the corresponding success rate Total Group data sample.

[0025] Subsequently, a discriminator is constructed based on a neural network with a gated recurrent unit (GRU). The goal of this discriminator is to learn from state trajectories. To the success rate of the task The mapping relationship. Discriminator The input is the state trajectory The state at each time step It needs to undergo normalization. The core of the network is a multi-layer GRU network used to encode temporal dependencies:

[0026] in, Indicates network parameters; The hidden state at the last time step is represented by this value, and its output after passing through a fully connected layer represents the prediction success rate. .

[0027] Discriminator The training process is a supervised learning process, using mean squared error as the loss function to measure the success rate of prediction. Compared to the true success rate label Differences between them:

[0028] The discriminator is trained using the Adam optimizer with the goal of minimizing the loss function. The trained discriminator is shown below. Based on the given state trajectory, a reliable estimate of the success rate of executing subsequent tasks in the initial state can be given as part of the reward and constraint function in step 3.

[0029] Step 2: Using the current observation matrix after the robot task switch, a dual-branch predictive encoder based on variational autoencoder is used to predict the robot's future state and the height change of the surrounding environment.

[0030] First, based on the current and past nine time-step ontological observation data, a historical observation input matrix is ​​constructed as the input to the predictive encoder. Then, a bi-branch predictive encoder based on a variational autoencoder is constructed to predict the robot's state at the next time step and the change in height of the surrounding environment, respectively. A resampling mechanism and regularization constraints are designed to limit overfitting. This part aims to construct a bi-branch predictive encoder and enable it to implicitly predict future changes in various factors based on the current state, thereby assisting the control strategy in making better decisions.

[0031] During training, the robot starts from an initial state of contact with the high platform and continuously interacts with the environment to generate observational data. Privileged observation information is equivalent to the data generated in step 1. It contains ontological observation data. Ontology observation data Including speed commands Robot angular velocity Robot gravity vector The position of each joint of the robot The speed of each joint of the robot and the action at the previous moment The data comprises 45 dimensions. During training, a historical observation matrix is ​​formed by taking the observations from the current moment and its nine nearest past time steps. This is used as the input part of the dual-branch predictive encoder to reflect the robot's recent state.

[0032] Based on the above inputs, the core of this step lies in constructing a two-branch prediction network based on a variational autoencoder. Unlike traditional deterministic encoders, variational autoencoders do not directly output implicit variables, but rather determine their probability distribution and then resample from the distribution to reconstruct the implicit variable. Regularization constraints are also introduced during the process, giving the two-branch prediction encoder stronger generalization ability.

[0033] dual-branch predictive encoder feedforward section Composed of a multi-layer MLP network, historical observation matrix As its input, its output is ,in, , Indicates the feedforward section of the dual-branch predictive encoder The latent spatial distribution mean and variance are output, and then latent variables are obtained by resampling from them. .

[0034] In the feedforward section Based on this, two neural networks with dual output heads are constructed, which share latent variables. As input, the first output head is the quadruped robot's future state prediction encoder. The output is ,in This represents the robot's predicted future state, used to fit the actual next time step state observed during training. The loss function is defined as the mean square error of both:

[0035] The second output head is an encoder for predicting height changes in the quadruped robot's surrounding environment. Its output is The output contains 12 elements, that is... It indicates that the center is the robot's center of mass. The height operation of the 12 sampling points in three rows and four columns evenly distributed within the region at the next time step.

[0036] This embodiment uses a variational encoder to enhance the generalization of the predictor variable. The purpose of the dual-branch is to decouple information with different meanings from the output. Otherwise, if a single output head is used to output the next state and the height change in parallel, it will lead to tight coupling between the two types of data with different meanings and magnitudes, which will easily cause deviations and mutual influence.

[0037] Traditional methods often employ similar approaches to acquire height data around a robot, thereby aiding control strategy decision-making. However, these methods mostly involve directly regressing specific height values ​​from the surrounding environment. During training, the model tends to precisely memorize the correspondence between complex terrain and height values ​​in the training data, rather than learning general patterns to discern height changes. Therefore, predictions may produce significant errors when faced with observation combinations not present in the training set. Furthermore, since height is a continuous variable, directly fitting it is highly sensitive to noise; even small input perturbations can cause drastic changes in the predicted value, thus misleading the control strategy.

[0038] Therefore, in this embodiment, when fitting the elevation data around the robot, the goal is no longer to precisely regress specific height values, but rather to predict the discrete height changes at each point. Specifically, for each point i, based on the actual height changes in the privileged information... Define discrete labels It is represented as:

[0039] For each point, the bi-branch predictive encoder predicts the output. This becomes the predicted probability for the three labels, i.e. The loss function is defined as the probability cross-entropy:

[0040] For a two-branch predictive encoder, the total loss function is:

[0041] in, This represents the probability that the i-th elevation point is predicted to be labeled w, and the preceding indicator function... Indicates only within the current tag It becomes 1 when it is a w tag, otherwise it is 0.

[0042] The last term of the total loss function is the KL divergence regularization constraint term for the VAE encoder, used to constrain the output distribution to approximate the prior distribution. , Indicates the batch quantity. This represents the adjustable divergence weight. This constraint encourages the bi-branch predictive encoder to generate different latent variables, thereby improving the generalization ability of the bi-branch predictive encoder.

[0043] Step 3: Utilize a deep reinforcement learning model to obtain the robot's current transitional control strategy based on the predicted future state of the robot and the height changes of the surrounding environment, so as to control the robot accordingly using the current transitional control strategy.

[0044] In this embodiment, a deep reinforcement learning model is constructed, and reward and constraint functions are set. The PPO algorithm is used for training. The deep reinforcement learning model adopts an architecture containing one policy network and two evaluation networks. The policy network is used to output the actions to be performed by the robot, and the two evaluation networks are used for reward evaluation and constraint evaluation, respectively.

[0045] The reward function includes multi-directional velocity following, success rate guidance, next policy value function, encouragement of bipedalism, forefoot height control, and torque constraint. The constraint functions include joint limits, maximum elevation angle limit, maximum joint velocity limit, and minimum success rate limit. Based on this, the PPO algorithm is used to optimize the transition strategy. After multiple iterations, the intermediate transition strategy learns to guide the robot towards the optimal intermediate transition state. This section aims to complete the construction of the main network structure required for reinforcement learning and to define in detail the reward and constraint functions used in the training process of the transition strategy.

[0046] Policy network is denoted as Current ontology observation Robots predict future states and the predicted elevation change matrix Together they form the overall observation matrix, which the policy network uses as input and outputs the actions the robot needs to perform. The two evaluation networks are denoted as reward evaluation networks. With constraint evaluation network They all use privileged observation As input, but with independent parameters and The value of the current state and the degree of constraint compliance are evaluated based on the reward function and the constraint function, respectively.

[0047] The reward function includes speed-following rewards. Forefoot height reward Standing on both feet as a guide and reward Torque Reward Success rate-guided rewards and the weighted value reward of the target control strategy As shown below: Speed ​​Follow Rewards To encourage the robot to track the desired linear and angular velocities, an exponential form is used to smooth them, which can be expressed as:

[0048] in, , , This represents the given command for the axial velocity along the x and y axes and the angular velocity along the z axis. , , This indicates the robot's current actual speed.

[0049] Forefoot height bonus This is used to guide the robot to autonomously raise its forelegs during the transition process, encouraging the robot to attach to or conform to the target high platform, and can be represented as:

[0050] in, This indicates the height of the robot's two front legs. This indicates the height of the platform the robot faces. This reward has a relatively small weight and is not a primary guiding factor, in order to avoid limiting the robot's forelegs to a fixed height.

[0051] Standing on both feet as a reward Objective and forefoot height reward Consistency, aimed at guiding the robot to tilt backward and adjusting the robot's current state to execute the target control strategy, can be represented as:

[0052] in, This indicates the height of the robot's two front legs. Indicates the height of the robot's hind legs. This indicates an indicator function, with an output of 0 or 1. This function aims to guide the robot's forelegs to be higher than its hind legs, preparing for the execution of the target control strategy.

[0053] Torque bonus The purpose is to limit the output torque of the robot joints, thereby protecting the hardware and reducing energy consumption, which can be expressed as:

[0054] in, This represents the torque value output by each joint at the current time step.

[0055] Task success rate guidance reward The core reward of the over-control strategy aims to guide the robot towards the best achievable state, ensuring optimal success rate when performing the target task after switching the target control strategy. This is achieved using the discriminator pre-trained in step 1. Based on the current state trajectory, predict the success probability of executing subsequent tasks. Add it to the reward items, represented as:

[0056] Here, 0.6 represents the critical success rate. Above this value, the reward value increases exponentially, strongly guiding the strategy to optimize for the success rate. This success rate-based reward setting method avoids the robot using a fixed posture as its goal, instead encouraging it to explore the optimal posture that can be safely reached.

[0057] Weighted value reward of target control strategy The goal is to make the state value portion of the transitional control policy approximate the target control policy. The output of the value evaluation network obtained during the training of the target control policy is... Introducing a reward item, represented as:

[0058] in, The attenuation coefficient of the objective value function is a hyperparameter that needs to be adjusted.

[0059] This embodiment introduces the value evaluation network from the target control strategy into the training framework of the transition control strategy. Specifically, the value evaluation network outputs the corresponding value assessment value based on the robot's current state, and incorporates it into the input or optimization objective of the transition control strategy through weighted fusion, thereby suppressing excessive influence on the transition strategy. The state value function of the transition control strategy exhibits partial convergence with the target control strategy. The mechanism is that the value assessment value generated by the target control strategy is introduced into the evaluation system of the transition control strategy through weighting, creating a positive correlation constraint between the two. During training, the transition control strategy aims to maximize its own cumulative value assessment value (i.e., the sum of reward functions), and the target strategy value assessment value contained within it is also maximized simultaneously. This achieves a local approximation of the transition control strategy to the target control strategy while preserving the differences between the two.

[0060] The overall reward function is expressed as speed-following reward. Forefoot height reward Standing on both feet as a guide and reward Torque Reward Success rate-guided rewards and the weighted value reward of the target control strategy sum.

[0061] The constraint function is used to define the safety boundary and includes joint limit constraints. Robot elevation angle constraint Joint velocity constraints Minimum success rate constraint As shown below: Joint limiting constraints The purpose is to limit the range of motion of the 12 joints of a quadruped robot to prevent the robot from adopting dangerous postures. It is defined as follows:

[0062] in, This indicates the current position of the j-th joint. , This represents the upper and lower limit values ​​of the j-th joint.

[0063] Robot elevation angle constraint The purpose is to limit the robot's forward and backward tilt to prevent it from tipping over due to excessive tilting along the y-axis. It is defined as follows:

[0064] in, This indicates the robot's current tilt angle along the y-axis in the body coordinate system. This indicates the maximum permissible tilt angle.

[0065] Joint velocity constraints The purpose is to limit the rotational speed of each joint of the robot, prevent dangerous actions, and reduce energy consumption. It is defined as follows:

[0066] in, This indicates the current rotational speed of the j-th joint. This indicates the maximum speed limit of the j-th joint.

[0067] Minimum success rate constraint The purpose is to ensure that the robot's final state, guided by the transition control strategy, is such that the success rate of switching to the target control strategy and executing the target task in this state is not lower than a set minimum value, defined as:

[0068] in, This indicates the success rate of the target task output by the success rate discriminator based on the current robot state, with 0.4 being the set minimum success rate.

[0069] The overall constraint function is expressed as joint limit constraint. Robot elevation angle constraint Joint velocity constraints Minimum success rate constraint sum.

[0070] Rewards and Evaluation Network The objective is to estimate the expected value of the cumulative reward under the current control policy and constrain the evaluation network. The objective is to calculate the expected value of the cumulative constraint violation rate under the current control strategy. During training, in each iteration, the robot starts from the initial state distribution generated by the initial strategy, interacts with the environment using the current transitional control strategy, and collects a state trajectory containing information such as observations, actions, reward values, and constraint values. Based on this trajectory, the reward evaluation network is calculated. Constraint Evaluation Network For each state's value estimation, the reward advantage function is derived using the generalized advantage estimation method. With constraint advantage function .

[0071] To ensure that the two types of advantage functions have consistent scaling, they are both normalized to obtain a normalized reward advantage function. With normalized constraint advantage function The sum and average of the two yields the comprehensive advantage function. Therefore, the policy network optimization loss function can be obtained as follows:

[0072] This loss function is the standard PPO-Clip objective function. Indicates the current strategy. This indicates the strategy before the update. This represents the pruning hyperparameter, used to limit the policy update step size. This represents the mean function.

[0073] Subsequently, the reward evaluation network was updated. Constraint Evaluation Network The loss functions are as follows, with the objective of minimizing the mean squared error between the estimated values ​​and the actual reward and constraint values:

[0074] in, , This represents the estimated reward and constraint values. , This represents the cumulative true value of the discount between reward and constraint rewards.

[0075] During training, the parameters of the policy network, reward evaluation network, constraint evaluation network, and bi-branch predictive encoder are updated together, and the total loss function is as follows:

[0076] Ultimately, through multiple rounds of iterative optimization, the transition control strategy can fully integrate various guidance methods, enabling the robot to complete adaptive state transitions under uncertain terrain, laying the foundation for a smooth switch of subsequent target control strategies.

[0077] Step 4: Construct dynamic weights corresponding to the initial control strategy, transition control strategy, and target control strategy respectively. After weighted fusion of the robot actions corresponding to the initial control strategy, transition control strategy, and target control strategy respectively, the robot's motion action is obtained. Among them, the corresponding dynamic weights are gradually increased to 1 according to the task stage of the robot, and the sum of the dynamic weights is kept at 1.

[0078] After training, a smooth switching of strategy outputs is achieved during strategy deployment. When the process reaches the point where a control strategy switch is required, the outputs of the initial, transitional, and target control strategies are dynamically weighted and fused before being executed by the robot. The weights are smoothly transitioned according to the task stage and remain summed to 1. The weight of the strategy to be executed in the current stage will eventually increase to 1. The purpose of this step is to enable a smoother switching between the three control strategies—initial, transitional, and target—during actual deployment, preventing abrupt changes in the robot's state space caused by sudden control strategy switching, which could lead to task failure.

[0079] This embodiment employs a dynamic weighted fusion strategy switching mechanism. The three control strategies—initial control strategy, transitional control strategy, and target control strategy—are as follows: , , During the control strategy switching period, for the robot's observation at a certain moment, the three strategies output actions respectively. , , At this stage, the robot does not simply execute a single control strategy to output an action, but rather a weighted sum of three sets of actions. Specifically, a dynamic weight vector is constructed. and satisfy The actual actions performed by the robot for:

[0080] In practical applications, the dynamic change of weights is determined by the task phase. In the initial phase, before the switching strategy command is given, then... The remaining weights are 0. When the robot begins to approach the high platform, a transition command is manually given, and the robot switches from the initial control strategy to the transition control strategy. Then, within 100 time steps... Linear interpolation from 1 to 0, Interpolated from 0 to 1, The value is always 0; once the transition strategy guides the robot to an ideal ready state, it automatically issues a command to switch to the target control strategy, still within 100 time steps. Linear interpolation from 1 to 0, Interpolated from 0 to 1, The value remains at 0, thus completing the switch and achieving the goal of leaping to a higher platform.

[0081] By constructing, training, and deploying the aforementioned transition strategies, quadruped robots can smoothly switch from control strategies that guide regular movements to control strategies that guide large-amplitude target movements, thereby improving the continuity of robot movement and the success rate of the robot in completing multi-stage tasks.

[0082] like Figure 1 As shown in this embodiment, the transition control method for assisting robots in switching control strategies includes the following steps in actual operation: Step S1: Run the initial strategy until the robot contacts the high platform, sample the state trajectory and task execution data, and train the GRU success rate discriminator to evaluate the completion of subsequent tasks; Step S2: Construct a future trend prediction module based on variational autoencoder to predict the next state of the robot dog and the height changes of the surrounding environment, and assist the strategy in making decisions; Step S3: Construct a policy network and a reward and constraint evaluation network, design multivariate reward and constraint functions, and then optimize the control strategy based on the PPO algorithm so that it can control the quadruped robot to reach the optimal transition state. Step S4: During deployment, the outputs of the three control strategies—target strategy, transition strategy, and control strategy—are dynamically weighted and merged. The weights change smoothly with the task stage, achieving natural switching between strategies.

[0083] This embodiment introduces a guidance mechanism based on the success rate of the target task execution, which enables the transition strategy to clearly perceive the impact of the current state on subsequent tasks. This effectively alleviates the problem that the robot is prone to failure when directly switching control strategies at uncertain states, and improves the reliability of the robot in completing complex multi-stage tasks.

[0084] This embodiment simultaneously predicts the robot's future trajectory and the discrete height changes of surrounding sampling points, enabling the transition strategy to guide the robot to adjust its posture in advance according to the trend of environmental changes. This effectively solves the problems of overfitting and poor generalization of traditional regression prediction methods.

[0085] This embodiment employs a dual-evaluation network to achieve specialized evaluation of rewards and constraints, avoiding conflicts in multi-objective optimization caused by a single evaluator. At the same time, it ensures that while maximizing rewards to complete the expected task, the strategy also strictly adheres to the constraints of the evaluator, limiting the occurrence of dangerous actions. The reward evaluation network focuses on maximizing task benefits, while the constraint evaluation network independently monitors the safety boundary, enabling the strategy to efficiently complete the task while strictly avoiding dangerous actions, thus improving the stability of the strategy.

[0086] Example 2 The purpose of this embodiment is to provide a transition control system that assists in switching control strategies for robots, including: The prediction module is configured to: use the current observation matrix after the robot task switch, and predict the robot's future state and the height change of the surrounding environment through a dual-branch predictive encoder based on a variational autoencoder. The transition control module is configured to: use a deep reinforcement learning model to obtain the robot's current transition control strategy based on the predicted future state of the robot and the height changes of the surrounding environment, and then use the current transition control strategy to control the robot accordingly; wherein, the deep reinforcement learning model includes a policy network and a dual evaluation network, determines the expected value of the cumulative reward under the current transition control strategy based on the reward evaluation network, determines the expected value of the cumulative constraint violation degree under the current transition control strategy based on the constraint evaluation network, and trains the current policy network and the current dual evaluation network based on the expected value of the cumulative reward and the expected value of the cumulative constraint violation degree until the parameters of the policy network and the dual evaluation network both meet the preset stability conditions, and determines the corresponding current transition control strategy for the robot based on the latest current policy network.

[0087] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0088] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0089] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0090] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0091] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0092] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0093] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0094] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0095] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0096] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A transition control method for assisting a robot in switching control strategies, characterized in that, include: Using the current observation matrix after the robot task switch, a bi-branch predictive encoder based on variational autoencoder is used to predict the robot's future state and the height change of the surrounding environment, respectively. The robot's future state and the height changes of its surrounding environment are predicted using a deep reinforcement learning model to obtain the robot's current transition control strategy, which is then used to control the robot accordingly. The deep reinforcement learning model includes a policy network and a dual evaluation network. The expected value of the cumulative reward under the current transition control policy is determined by the reward evaluation network, and the expected value of the cumulative constraint violation degree under the current transition control policy is determined by the constraint evaluation network. The current policy network and the current dual evaluation network are trained based on the expected value of the cumulative reward and the expected value of the cumulative constraint violation degree until the parameters of the policy network and the dual evaluation network both meet the preset stability conditions. The corresponding current transition control policy of the robot is determined based on the latest current policy network. Dynamic weights are constructed for the initial control strategy, the transition control strategy, and the target control strategy, respectively. The robot actions corresponding to the initial control strategy, the transition control strategy, and the target control strategy are weighted and fused to obtain the motion action executed by the robot. The dynamic weights are gradually increased to 1 according to the task stage in which the robot is located, and the sum of the dynamic weights is kept at 1.

2. The transition control method for assisting a robot in switching control strategies as described in claim 1, characterized in that, Also includes: Based on the robot's state trajectory after switching target control strategies, the trained discriminator is used to obtain the task success rate of executing subsequent tasks under the current state trajectory, and the task success rate is added to the cumulative reward calculation of the reward evaluation network. Specifically, the discriminator is trained based on the robot's state trajectory and task execution status after the robot switches target control strategies, so that the discriminator learns the mapping relationship between the robot's state trajectory and the task execution success rate.

3. The transition control method for assisting a robot in switching control strategies as described in claim 1, characterized in that, The loss function for training the bi-branch predictive encoder based on variational autoencoder includes a mean squared error function, a probabilistic cross-entropy function, and a KL divergence regularization constraint; wherein, the mean squared error function is determined based on the robot's predicted future state and the observed actual next time step state; and the probabilistic cross-entropy function is determined based on the discrete labels corresponding to the predicted discrete height changes and the actual height changes of each point.

4. The transition control method for assisting a robot in switching control strategies as described in claim 1, characterized in that, A reward function is created so that the reward evaluation network can use the reward function to determine the expected value of the cumulative reward under the current transition control strategy; the reward function includes speed following reward, forefoot height reward, bipedal standing guidance reward, torque reward, task success rate guidance reward, and weighted value reward of the target control strategy.

5. The transition control method for assisting a robot in switching control strategies as described in claim 1, characterized in that, Create constraint functions so that the constraint evaluation network can use the constraint functions to determine the expected value of the cumulative constraint violation degree under the current transition control strategy; wherein, the constraint functions include joint limit constraints, robot elevation angle constraints, joint velocity constraints, and minimum success rate constraints.

6. A transition control system for assisting a robot in switching control strategies, characterized in that, include: The prediction module is configured to: use the current observation matrix after the robot task switch, and predict the robot's future state and the height change of the surrounding environment through a dual-branch predictive encoder based on a variational autoencoder. The transition control module is configured to: utilize a deep reinforcement learning model to obtain the robot's current transition control strategy based on the predicted future state of the robot and changes in the surrounding environment, and then use this strategy to control the robot accordingly; wherein, the deep reinforcement learning model includes a policy network and a dual evaluation network, determines the expected value of the cumulative reward under the current transition control strategy based on the reward evaluation network, determines the expected value of the cumulative constraint violation degree under the current transition control strategy based on the constraint evaluation network, and trains the current policy network and the current dual evaluation network based on the expected values ​​of the cumulative reward and the cumulative constraint violation degree until the parameters of both the policy network and the dual evaluation network meet preset stability conditions, and determines the corresponding current transition control strategy for the robot based on the latest current policy network; construct dynamic weights corresponding to the initial control strategy, transition control strategy, and target control strategy respectively, and then weight and fuse the robot actions corresponding to the initial control strategy, transition control strategy, and target control strategy respectively to obtain the motion action executed by the robot; wherein, the corresponding dynamic weights are gradually increased to 1 according to the task stage in which the robot is, and the sum of the dynamic weights remains at 1.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Images