Event-triggered and world model-based control method and system for legged robots

Abstract

The present application relates to the field of robot control and artificial intelligence technology, in particular to a legged robot control method and system based on event triggering and world model, the method comprises: obtaining real-time observation data of the legged robot; the real-time observation data comprises: visual observation data and proprioceptive perception data; inputting the real-time observation data into the trained control model, the trained control model processes the real-time observation data based on event triggering and world model to obtain the action instruction of the robot, and sending the action instruction of the robot to the controller, and the controller sends the instruction to the joint motor to complete the control of the legged robot; the present application actively triggers perception only when the environment mutates, and still has high robustness under visual loss.

Description

Technical Field

[0001] This invention relates to the fields of robot control and artificial intelligence technology, and in particular to a control method and system for legged robots based on event triggering and world models. Background Technology

[0002] Early legged robot control primarily relied on explicit physical modeling methods such as model prediction. However, in complex, unstructured environments, control often failed due to mismatch between environmental parameters and the model. To overcome this deficiency, model-free deep reinforcement learning gradually became mainstream. However, this method not only has extremely low sample efficiency, but also easily leads to onboard computational overload and model overfitting when directly processing high-dimensional visual observations. To address the aforementioned state representation and computational bottlenecks, recent research has introduced a "world model" architecture, enabling robots to learn environmental dynamics and plan actions in a low-dimensional latent space. However, existing world model-based perception methods still have a key technical flaw: they mainly rely on a fixed-frequency update mechanism triggered by "time." That is, regardless of whether the environment changes, the system rigidly calls the high-computing-power visual encoder at a fixed cycle. This rigid strategy leads to a double dilemma: on the one hand, on flat and predictable terrain, frequent visual processing results in serious waste of computational power and energy consumption; on the other hand, in abrupt terrain changes (such as suddenly appearing deep pits) or high-speed movement, fixed low-frequency updates can easily cause the robot to miss key dangerous features, leading to safety accidents such as missteps and falls.

[0003] Furthermore, while existing world-model-based methods possess some state extrapolation capabilities, their scheduling of visual perception resources remains rigid and open-loop. Specifically, existing systems cannot assess their real-time understanding of environmental conditions (i.e., they cannot determine whether there are discrepancies between the terrain predicted in their minds and the actual environment). Therefore, they can only mechanically process visual information at a fixed frequency, failing to achieve an "on-demand perception" mechanism that allows for proactive visual dormancy to conserve computing power in flat terrain and rapid activation of vision in complex and abrupt terrain. Simultaneously, if visual sensors unexpectedly fail or experience signal interruption due to external interference (such as smoke, strong light, or hardware malfunction), existing control strategies that heavily rely on continuous visual input are highly susceptible to failure, generally lacking robust mechanisms to maintain balance even in visually impaired (blindly walking) states. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a legged robot control method and system based on event triggering and a world model. The core innovation of this invention lies in proposing a legged robot control framework that combines an event triggering mechanism based on prior prediction errors with a modal random dropout training strategy. This invention is a legged robot motion perception and control method that deeply couples the internal prior prediction mechanism with the external sensor triggering logic, enabling proactive perception triggered only when the environment undergoes abrupt changes, and maintaining high robustness even in the absence of visual input. Unlike existing technologies that rely on fixed-frequency time-triggered mechanisms (i.e., blindly calling visual perception at fixed intervals regardless of terrain), resulting in wasted computational power and response delays, this invention utilizes the prior prediction capability of a world model to determine whether a sudden change in the environment has occurred before visual encoding, achieving "on-demand perception." Simultaneously, it combines a hybrid attention mechanism to enhance the noise resistance of feature extraction and introduces a modal random dropout training strategy to solve the stability problem of the robot's "blind walking" during visual silence periods.

[0005] On the one hand, a control method for legged robots based on event triggering and a world model is provided, including:

[0006] Acquire real-time observation data of the legged robot; the real-time observation data includes: visual observation data and proprioceptive perception data;

[0007] Real-time observation data is input into the trained control model. The trained control model processes the real-time observation data to obtain the robot's motion commands. The robot's motion commands are then sent to the controller, which in turn sends the commands to the joint motors to complete the control of the legged robot.

[0008] The trained control model processes real-time observation data in the following ways: if it is the first execution, it processes both the visual observation data and the proprioceptive perception data at the current moment to obtain the robot's action command; if it is not the first execution, the trained control model makes prior predictions based on the world model, first determining whether the environment has changed abruptly. If a change occurs, it determines the robot's next motion strategy based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no change occurs, it determines the robot's next motion strategy based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model.

[0009] On the other hand, a legged robot control system based on event triggering and a world model is provided, including:

[0010] The acquisition module is configured to acquire real-time observation data of the legged robot; the real-time observation data includes visual observation data and proprioceptive perception data.

[0011] The output module is configured to: input real-time observation data into the trained control model, process the real-time observation data to obtain the robot's motion commands, send the robot's motion commands to the controller, and then send the commands to the joint motors to complete the control of the legged robot.

[0012] The trained control model processes real-time observation data in the following ways: if it is the first execution, it processes both the visual observation data and the proprioceptive perception data at the current moment to obtain the robot's action command; if it is not the first execution, the trained control model makes prior predictions based on the world model, first determining whether the environment has changed abruptly. If a change occurs, it determines the robot's next motion strategy based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no change occurs, it determines the robot's next motion strategy based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0014] 1. Achieved high computational efficiency based on "prior prediction": Existing technologies typically require running a visual convolutional network to extract visual features before calculating the posterior error, causing the visual encoder to run at full capacity at all times, failing to truly save onboard computing power. This invention innovatively utilizes the prior prediction capability of the world model, with an external lightweight decoder generating extremely low resolution (e.g., The system generates a priori prediction image (pixels). Before activating the high-computational visual encoder, future observations are predicted solely through a low-computational recurrent neural network. Visual perception is activated only when the deviation between the priori prediction and the downsampled real observation exceeds an adaptive threshold. This mechanism allows the robot to completely skip the visual encoding step on flat terrain, significantly reducing the system's floating-point computation at the source and substantially extending the robot's operational endurance.

[0015] 2. This invention addresses the control instability problem during "perceptual silence" (blind zone robustness): Traditional event-triggered control is prone to severe jitter or drift in the policy network output due to input loss during sensor shutdown (silent mode). This invention introduces a modal random dropout training strategy, randomly blocking visual input during the training phase, forcing the reinforcement learning strategy to learn to "walk blindly" using proprioception and historical memory. This allows the robot to maintain extremely high motion stability even in long-term visual silence modes or in extreme environments where smoke or strong light cause visual impairment, preventing missteps or falls.

[0016] 3. Enhanced feature extraction accuracy in complex dynamic environments: Addressing the common challenges faced by legged robots, such as complex ground textures and drastic lighting changes, this invention integrates a hybrid attention mechanism at the feature extraction stage. Through the synergistic effect of channel attention and spatial attention, the system can automatically suppress high-frequency noise such as water stains and shadows on the ground, and concentrate computational resources on key geometric features such as step edges and ditch boundaries. This ensures that even with low-frequency updates, each triggered visual update provides a high-quality, highly saliency representation of the environment. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is a flowchart of the method in Example 1. Detailed Implementation

[0019] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] Example 1

[0021] like Figure 1 As shown, this embodiment provides a legged robot control method based on event triggering and a world model, including:

[0022] S101: Acquire real-time observation data of the legged robot; the real-time observation data includes: visual observation data and proprioceptive perception data;

[0023] S102: The real-time observation data is input into the trained control model. The trained control model processes the real-time observation data to obtain the robot's motion commands, which are then sent to the controller. The controller then sends the commands to the joint motors to complete the control of the legged robot. The processing of real-time observation data by the trained control model includes: if it is the first execution, both the visual observation data and the proprioceptive perception data at the current moment are processed to obtain the robot's motion commands; if it is not the first execution, the trained control model performs prior prediction based on the world model to first determine whether a sudden change has occurred in the environment. If a sudden change has occurred, the robot's next motion strategy is determined based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no sudden change has occurred, the robot's next motion strategy is determined based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model.

[0024] Furthermore, the visual observation data includes: real-time high-resolution depth images acquired by an airborne depth camera; the body perception data includes: the body angular velocity of the legged robot, the projected gravity vector (used to characterize the body posture), the target velocity command, the joint angles of each leg joint, the joint angular velocities of each leg joint, and the joint movements executed by the robot in the previous control cycle.

[0025] After acquiring the high-resolution depth image, it is immediately downsampled and preprocessed using average pooling to filter out high-frequency noise and extract a low-resolution true depth image that reflects the general contours of the terrain (e.g., ...). Pixel compression to (pixels), this low-resolution true depth image will serve as a baseline for subsequent calculation of prior prediction error.

[0026] Furthermore, the control model includes: a visual encoder, a proprioceptive encoder, and a world model;

[0027] The visual encoder is used to extract visual features from visual observation data.

[0028] The body perception encoder is used to extract features from the body perception data of the legged robot to obtain body perception features.

[0029] The world model is used to determine whether a sudden change has occurred in the environment based on prior inference capabilities and to generate a switch signal. The switch signal is used to control the wake-up and sleep of the visual encoder and to complete the hidden state update.

[0030] Visual features are input into a multiplier, which also receives a switch signal. If the switch signal is equal to 1, the visual encoder is activated, and the multiplier outputs the actual visual features. If the switch signal is equal to 0, the visual encoder remains in a dormant state, and the multiplier outputs a zero feature vector with the same dimension as the visual features. The initial switch signal is equal to 1. The multiplier is used to calculate the product of the visual features and the switch signal.

[0031] The output of the multiplier is combined with the ontology-aware features through the first splicing unit to obtain the observed features;

[0032] The observed features are input into the world model. Based on its prior prediction capabilities, the world model determines whether a sudden change has occurred in the environment. If a sudden change occurs, the switch signal is set to 1, and the next movement strategy is determined based on the current visual features, the proprioceptive features, and the deterministic cyclic state at the current moment deduced from the world model. If no sudden change occurs, the switch signal is set to 0, and the next movement strategy is determined based on the current proprioceptive features and the deterministic cyclic state at the current moment deduced from the world model.

[0033] Furthermore, the world model, based on its prior predictive capabilities, determines whether a sudden change has occurred in the environment, specifically including:

[0034] The gated cyclic unit network of the world model outputs the deterministic cyclic state at the current moment based on the historical state of the previous moment. The deterministic cyclic state at the current moment is input into the prior probability distribution network of the world model, and the prior probability distribution network of the world model outputs the predicted prior random state.

[0035] The predicted prior random state is concatenated with the current deterministic cyclic state to obtain the concatenated state. The concatenated state is then input into the prior decoder, which generates a prior predicted low-resolution depth image.

[0036] The error calculation unit calculates the prior prediction error between the prior predicted low-resolution depth image and the true low-resolution depth image obtained after downsampling preprocessing;

[0037] The judgment unit compares the prior prediction error with the adaptive triggering threshold at the current time to obtain the switch signal at the current time. The switch signal at the current time is sent into the multiplier. The switch signal equal to 1 indicates that the environment has changed abruptly, and the switch signal equal to 0 indicates that the environment has not changed abruptly.

[0038] Further, the judgment unit compares the prior prediction error with the adaptive triggering threshold at the current time to obtain the switching signal at the current time, including:

[0039] If the prior prediction error is greater than the adaptive trigger threshold at the current moment, it indicates that a sudden change in terrain has occurred, and the visual encoder is activated.

[0040] If the prior prediction error is less than or equal to the adaptive triggering threshold at the current time, it means that the terrain meets expectations, no environmental changes have occurred, and the visual encoder remains dormant.

[0041] Furthermore, based on current visual features, ontological perception features, and the deterministic cyclic state deduced from the world model at the current moment, the next motion strategy is determined, including:

[0042] The observed features are input into the posterior probability distribution network of the world model, which outputs the posterior state. The posterior state is then concatenated with the current deterministic cyclic state output by the gated recurrent unit network of the world model to obtain a first concatenation result. This first concatenation result is then concatenated again with the ontological perception data and external control commands to obtain a second concatenation result. This second concatenation result is input into the policy network (Actor), which outputs predicted action commands for the legged robot. Based on these predicted action commands, the robot drives the motors of each joint, updates the time step, and returns to the gated recurrent unit network of the world model to execute the next control cycle. The external control commands include the robot's desired linear velocity along the X-axis, the desired linear velocity along the Y-axis, and the desired angular velocity around the Z-axis.

[0043] Furthermore, based on the current ontology perception features and the deterministic cyclic state deduced from the world model at the current moment, the next motion strategy is determined, including:

[0044] The output value of the prior probability distribution network (i.e., the prior state) is used as the actual random state at the current moment. This actual random state is concatenated with the deterministic cyclic state at the current moment output by the gated cyclic unit network of the world model to obtain the third concatenation result. The third concatenation result is then concatenated again with the ontology perception data and external control commands to obtain the fourth concatenation result. The fourth concatenation result is input into the policy network Actor, which outputs the predicted action commands for the legged robot. Based on the predicted action commands, the robot drives the motors of each joint to work. Subsequently, the time step is updated, and the robot returns to the gated cyclic unit network of the world model to execute the next control cycle.

[0045] Furthermore, the visual encoder includes:

[0046] The system comprises a multi-layer convolutional neural network, a channel attention feature extraction layer, and a spatial attention feature extraction layer connected in sequence. The multi-layer convolutional neural network is used to extract high-dimensional implicit feature tensors. The channel attention feature extraction layer is used to identify key features. The spatial attention feature extraction layer is used to identify the spatial location of key features.

[0047] Furthermore, the channel attention feature extraction layer includes:

[0048] Input feature map Spatial information is aggregated using global max pooling and global average pooling, and then input into a multilayer perceptron (MLP) for feature mapping. In this embodiment, the MLP preferably contains two hidden layers (with, for example, 64 and 32 neurons respectively), and uses ReLU as the activation function to retain key channel information while reducing dimensionality. The output of the MLP is then summed and activated by a sigmoid algorithm to generate a channel attention map. The mathematical expression is:

[0049] ;

[0050] in, This represents the original feature map of the input (the specific source of this feature map is: a high-dimensional implicit feature tensor obtained by processing real-time depth images captured by an airborne depth camera and processed by a multi-layer convolutional neural network at the front end of the visual encoder). This represents the generated channel attention weight map; This represents the Sigmoid activation function; Indicates global average pooling. This represents the global max pooling operation. This represents a multilayer perceptron. Represents a real tensor or a multidimensional matrix. Represents the input feature map Spatial height, Represents the input feature map The width of the space.

[0051] Subsequently, the input feature map is multiplied by the channel weights to obtain the intermediate feature map after channel attention weighting. The calculation formula is:

[0052] ;

[0053] in, This indicates element-wise multiplication.

[0054] Furthermore, the spatial attention feature extraction layer includes:

[0055] feature map Max pooling and average pooling are performed along the channel dimension, followed by feature aggregation through a convolutional layer. Specifically, this convolutional layer uses a 7×7 kernel with a stride of 1 and 'same' padding to ensure that the output spatial attention map is the same size as the input feature map. The convolutional output is then activated by a sigmoid function to generate the spatial attention map. The mathematical expression is as follows:

[0056]

[0057] in, This represents the generated spatial attention weight map; Indicates the kernel size as Convolution operations; This indicates a splicing operation along the channel dimension. Indicates global average pooling. This represents the global max pooling operation. This represents the Sigmoid activation function;

[0058] Finally, highly saliency visual features are generated. , The calculation formula is:

[0059]

[0060] in, This represents highly saliency visual features, which are then used to input the world model. This represents the generated spatial attention weight map; This represents the intermediate feature map after channel attention weighting; This indicates element-wise multiplication.

[0061] Furthermore, the body perception encoder is implemented using a multilayer perceptron.

[0062] Furthermore, the world model includes: a gated recurrent unit network, a prior probability distribution network, and a posterior probability distribution network;

[0063] The input values ​​of the gated recurrent unit network are the deterministic state of the previous time step, the random latent state of the previous time step, and the robot's action vector of the previous time step; the output value of the gated recurrent unit network is the deterministic recurrent state of the current time step.

[0064] The input value of the prior probability distribution network is the deterministic cyclic state at the current moment, and the output value of the prior probability distribution network is the predicted prior random state.

[0065] The input value of the posterior probability distribution network is the concatenation result of the observed features and the deterministic cyclic state at the current time; the output value of the posterior probability distribution network is the corrected posterior random state.

[0066] The core inference engine is a world model architecture based on the Recurrent State Space Model (RSSM). The Recurrent State Space Model is a world model architecture that integrates cyclic memory and probabilistic state modeling. It is used to remember historical environmental states and predict future dynamics, providing a basis for event trigger determination.

[0067] Furthermore, the gated loop unit network includes:

[0068] The current moment is calculated by using a gated loop unit to memorize historical information. Deterministic cyclic state :

[0069]

[0070] in, It represents the deterministic state at the previous moment; This represents the random potential state at the previous time step; This represents the action vector executed by the robot in the previous moment; This represents a gated cyclic unit network.

[0071] The deterministic state of the previous moment is a historical memory feature vector maintained by the gated recurrent unit (GRU). It integrates all historical observations and actions of the robot in the previous moment and before, representing the system's deterministic contextual memory of the historical trajectory.

[0072] The random latent state of the previous moment is a feature vector that is sampled from the probability distribution and represents the implicit state of the robot and the environment at the previous moment.

[0073] Furthermore, the prior probability distribution network includes:

[0074] Predicting the current random state distribution based solely on historical information in the absence of visual input:

[0075]

[0076] in, Represents the predicted prior random state; Indicates by parameters Parameterized prior probability distribution network; Indicates the current time The random potential states; Indicates the current time A deterministic cyclic state. This indicates that a random variable follows a certain distribution.

[0077] Furthermore, the prior probability distribution network is specifically implemented using a multilayer perceptron (MLP) to output the statistical parameters of a multidimensional Gaussian distribution, and sampling is performed using a reparameterization technique. Its expanded calculation formula is as follows:

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] in, This represents the deterministic loop state at the current moment (i.e., the output of the gated loop unit). This represents the feature extraction layer of a multilayer perceptron in the prior network, which contains a fully connected layer, layer normalization, and activation function. This represents the hidden layer feature vector extracted by the multilayer perceptron. This represents the weight matrix of the linear mapping layer used to calculate the statistical parameters of the probability distribution; This represents the bias vector of the linear mapping layer used to calculate the statistical parameters of the probability distribution; This represents the mean vector of the prior Gaussian distribution of the prediction; This represents the original standard deviation vector of the network output; This represents the Softplus activation function, used to ensure that the standard deviation is a positive number; This represents the preset minimum standard deviation constant (to prevent calculation errors caused by zero variance); This represents the standard deviation vector of the final prior Gaussian distribution after nonlinear activation and boundary processing. Indicates the distribution from the standard normal distribution The noise vector obtained by random sampling; Represents the element-wise multiplication of vectors (Hadamard product). This represents the prior random state of the predicted final output.

[0083] To achieve low-computational-power event-triggered determination, a lightweight image decoder is used to directly generate the predicted observation image from the prior state. This process relies solely on historical extrapolation and does not require access to airborne depth camera data. The predicted image output by the decoder is aligned in size with the low-resolution ground truth image obtained from the first preprocessing step.

[0084] Furthermore, the decoder includes:

[0085]

[0086] in, Represents the prior predicted depth image ; Indicates the current time Deterministic cyclic state; Represents the predicted prior random state. express Parameterized image decoder network.

[0087] Furthermore, the image decoder network is implemented using a lightweight transposed convolutional architecture to reduce on-device computational power consumption. The image decoder network takes the prior features deduced from the previous time step as input and outputs a low-resolution predicted image through spatial reconstruction for rapid calculation of the trigger error. Its expanded calculation formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092] in, This represents the prior feature vector, derived from the deterministic cyclic state at the current time step. and the predicted prior random state It is pieced together; This represents a fully connected mapping layer, used to map one-dimensional feature vectors into high-dimensional vectors; This represents a tensor shape reconstruction operation, which transforms a one-dimensional vector into a vector with a number of channels. Initial height and width (For example 3D initial feature map ; This indicates the first layer of transposed convolution operation, used to upsample and amplify the feature map (e.g., amplifying it layer by layer to...). size); This indicates the second layer of transposed convolution operation, used to upsample and amplify the feature map (e.g., amplifying it layer by layer to...). size); This represents the exponential linear unit activation function, used to introduce nonlinear expressions; This represents the prior prediction image (mean of the low-resolution image) that the decoder finally outputs. This prediction image will be directly compared with the actual downsampled observation image to calculate the trigger error.

[0093] Furthermore, the posterior probability distribution network includes:

[0094]

[0095] in, This represents the corrected posterior random state; Indicates the current time Deterministic cyclic state; This represents the observed data at the current moment; Indicates by parameters Parameterized posterior probability distribution network.

[0096] Furthermore, the posterior probability distribution network is specifically implemented using a multilayer perceptron, which is used to fuse observed features and output the statistical parameters of the posterior multidimensional Gaussian distribution, and then perform sampling through reparameterization techniques. Its expanded calculation formula is as follows:

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] in, This represents the deterministic cyclic state at the current moment. The observed feature vector at the current time Perform the splicing operation; This represents the feature extraction layer of a multilayer perceptron in the posterior network (including fully connected layers, layer normalization, and activation functions). This represents the hidden layer feature vector extracted after fusing historical memory and current observation through a multilayer perceptron. These represent the weight matrices of the linear mapping layer used to calculate the distribution parameters in the posterior network; This represents the bias vector of the linear mapping layer used to compute the distribution parameters in the posterior network; Let represent the mean vectors of the predicted posterior Gaussian distribution, respectively. and represent the original standard deviation vectors of the predicted posterior Gaussian distribution, respectively; This represents the Softplus activation function, used to ensure that the standard deviation is a positive number; This represents the preset minimum standard deviation constant (to prevent calculation errors caused by zero variance); This represents the standard deviation vector of the final prior Gaussian distribution after nonlinear activation and boundary processing. Indicates the distribution from the standard normal distribution The noise vector obtained by random sampling; Represents the element-wise multiplication of vectors (Hadamard product). This represents the final output posterior random state, corrected for by actual observed features.

[0102] By lightweightly comparing the deviation between "the model's prediction of the future" and "the actual low-resolution environment", an adaptive dynamic threshold is used to determine whether to wake up the high-computing visual encoder.

[0103] Further, the error calculation unit calculates the prior prediction error between the prior predicted depth image and the current real depth image, including:

[0104] Calculate the low-resolution true depth image obtained by downsampling Compared with prior prediction of low-resolution depth images The mean square error between the two is defined as the prior prediction error at the current time. :

[0105]

[0106] in, Indicates the current time The prior prediction error; This represents a low-resolution depth image that was actually captured and downsampled at the current moment. This indicates that a priori prediction of a low-resolution depth image is generated using a decoder.

[0107] The error calculation here is performed only at extremely low resolution (such as 256 pixels). Although the true image overview of the current frame is read, the computational cost is minimal because the high-performance convolutional visual coding network has not yet been called to extract high-dimensional features.

[0108] Furthermore, the adaptive trigger threshold at the current moment includes:

[0109] To adapt to terrains of varying complexity, a fixed trigger threshold was abandoned, and an exponential moving average mechanism based on historical errors was adopted to dynamically generate an adaptive trigger threshold for the current moment. :

[0110]

[0111]

[0112] in, This represents the exponential moving average of the historical error at the current moment. The smoothing coefficient is (preferably 0.99 in this embodiment; the closer it is to 1, the smoother the threshold change). The basic sensitivity threshold (preferably 0.05 in this embodiment); The weights are adaptively adjusted (preferably 0.5 in this embodiment). This mechanism allows the robot to appropriately relax the triggering conditions in continuously rugged terrain to avoid computational overload, while maintaining extremely high sensitivity when transitioning from flat to rugged terrain. This represents the exponential moving average of the historical error at the previous moment. This represents the prior prediction error at the current moment.

[0113] Further, the judgment unit compares the prior prediction error with the adaptive triggering threshold at the current time to obtain the switching signal at the current time, including:

[0114] Based on the current error With adaptive threshold The comparison results are used to generate discrete control signals. :

[0115]

[0116] if If the terrain changes abruptly or does not develop as expected, the visual encoder is immediately activated to extract the high-resolution original image features of the current frame and input them into the world model for visual posterior correction.

[0117] if If the terrain is determined to be flat or fully as expected, the visual encoder is forced to remain in a dormant state (logically manifested as zeroing the visual feature vector mask), cutting off the visual processing path, and relying entirely on the prior inference state of the world model and the ontological perception data for subsequent motion control, thereby greatly saving onboard computing power and battery energy consumption.

[0118] Furthermore, the training process of the trained control model includes:

[0119] A training set is constructed, which consists of real-time observation data of a legged robot with a known walking strategy; wherein, the legged robot walking strategy includes: leg lifting, jumping, crawling, and side-stepping;

[0120] The training set is input into the control model to train the control model. Training is stopped when the total loss function value of the model no longer decreases, and the trained control model is obtained.

[0121] During training, the multiplier is disconnected from the decision unit, and the input of the multiplier is a random mask matrix that follows a Bernoulli distribution.

[0122] To ensure that the control policy does not cause instability or control collapse when visual features are forcibly reset to zero, a modality random dropout mechanism is introduced during the reinforcement learning policy training phase:

[0123] During the offline training phase, instead of relying on a priori prediction error triggering mechanism, a random mask matrix following a Bernoulli distribution is introduced. During each action sampling, the high-dimensional visual latent state features output by the world model are forcibly masked with a preset probability (preferably 20% in this embodiment) (i.e., the feature vector is multiplied by the mask to zero), while the input of ontology perception historical information is always retained.

[0124] During the offline training phase, the Actor-Critic policy network (where the Actor network is responsible for generating action instructions and the Critic network is responsible for evaluating the value of actions to assist in optimization) is forced to repeatedly adapt to the extreme condition of "sudden blindness" during training. This forces it to deeply mine and utilize the ontological perception historical memory (multi-frame ontological states processed by the historical encoder) to maintain the body's balance and infer the terrain, thus laying a solid foundation for robustness under "sparse perception".

[0125] Furthermore, the training and deployment process of the control model specifically includes:

[0126] In the offline training phase: A large-scale parallel simulation environment is constructed, and the legged robot's proprioception, terrain observation data, and reference motion capture dataset (which contains known legged robot walking strategies, such as leg lift, jumping, crawling, and sidestepping) are loaded into the environment. The control model is trained using the reinforcement learning-based Actor-Critic algorithm. The robot continuously interacts with the environment and samples real-time observation data in the simulation environment. The strategy is optimized by maximizing the cumulative reward function until the strategy converges or reaches the preset maximum number of iterations, resulting in the trained control model.

[0127] To ensure that the control policy does not become unstable or collapse when visual features are forcibly reset to zero (i.e., when the triggering mechanism determines that visual updates are unnecessary), a modality random dropout mechanism is introduced during the reinforcement learning policy training phase: During training, the multiplier is disconnected from the decision unit, and the input to the multiplier is replaced with a random mask matrix following a Bernoulli distribution. The random mask matrix following a Bernoulli distribution is a matrix containing only elements 0 and 1, and its generation probability follows a Bernoulli distribution.

[0128] Specifically, during each action sampling, a random mask matrix following a Bernoulli distribution forcibly masks the high-dimensional visual latent state features output by the world model with a preset probability (preferably 20% probability in this embodiment) (i.e., multiplying the feature vector by 0 in the mask matrix to achieve feature zeroing), while always retaining the input of ontology perception history information.

[0129] The Actor-Critic policy network (where the Actor network is responsible for generating action instructions and the Critic network is responsible for evaluating the value of actions to assist in optimization) is forced to repeatedly adapt to the extreme condition of "sudden blindness" during training. This forces it to deeply mine and utilize the proprioceptive historical memory (multi-frame proprio states processed by the historical encoder) to maintain body balance and infer terrain, thus laying a solid foundation for robustness under "sparse perception".

[0130] In the actual online deployment and control loop, the ontology perception features processed by the encoder, the target velocity command, and the hidden state of the world model processed by perception scheduling are first spliced ​​and fused along the channel dimension. Then, this fused feature is input into the Actor policy network. The Actor policy network outputs the control command (e.g., target joint angle) in real time, driving the leg joint motors of the legged robot to perform the corresponding movements and entering the next control loop.

[0131] Example 2

[0132] This embodiment provides a legged robot control system based on event triggering and a world model, including:

[0133] The acquisition module is configured to acquire real-time observation data of the legged robot; the real-time observation data includes visual observation data and proprioceptive perception data.

[0134] The output module is configured to: input real-time observation data into the trained control model, process the real-time observation data to obtain the robot's motion commands, send the robot's motion commands to the controller, and then send the commands to the joint motors to complete the control of the legged robot.

[0135] The trained control model processes real-time observation data in the following ways: if it is the first execution, it processes both the visual observation data and the proprioceptive perception data at the current moment to obtain the robot's action command; if it is not the first execution, the trained control model makes prior predictions based on the world model, first determining whether the environment has changed abruptly. If a change occurs, it determines the robot's next motion strategy based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no change occurs, it determines the robot's next motion strategy based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model.

[0136] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for controlling a legged robot based on event triggering and world model, characterized in that, include: Acquire real-time observation data of the legged robot; The real-time observation data includes: visual observation data and proprioceptive perception data; Real-time observation data is input into the trained control model. The trained control model processes the real-time observation data to obtain the robot's motion commands. The robot's motion commands are then sent to the controller, which in turn sends the commands to the joint motors to complete the control of the legged robot. The trained control model processes real-time observation data in the following ways: if it is the first execution, it processes both the visual observation data and the proprioceptive perception data at the current moment to obtain the robot's action command; if it is not the first execution, the trained control model makes prior predictions based on the world model, first determining whether the environment has changed abruptly. If a change occurs, it determines the robot's next motion strategy based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no change occurs, it determines the robot's next motion strategy based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model. The control model includes: a visual encoder, a proprioceptive encoder, and a world model; The visual encoder is used to extract visual features from visual observation data. The body perception encoder is used to extract features from the body perception data of the legged robot to obtain body perception features. The world model is used to determine whether a sudden change has occurred in the environment based on prior inference capabilities and to generate a switch signal. The switch signal is used to control the wake-up and sleep of the visual encoder and to complete the hidden state update. Visual features are input into a multiplier, which also receives a switch signal. If the switch signal is equal to 1, the visual encoder is activated, and the multiplier outputs the actual visual features. If the switch signal is equal to 0, the visual encoder remains in a dormant state, and the multiplier outputs a zero feature vector with the same dimension as the visual features. The initial switch signal is equal to 1. The multiplier is used to calculate the product of the visual features and the switch signal. The output of the multiplier is combined with the ontology-aware features through the first splicing unit to obtain the observed features; The observed features are input into the world model. Based on its prior prediction capabilities, the world model determines whether a sudden change has occurred in the environment. If a sudden change has occurred, the switch signal is set to 1, and the next movement strategy is determined based on the current visual features, the proprioceptive features, and the deterministic cyclic state at the current moment deduced from the world model. If no sudden change has occurred in the environment, the switch signal is set to 0, and the next movement strategy is determined based on the current proprioceptive features and the deterministic cyclic state at the current moment deduced from the world model. World models, based on their prior predictive capabilities, determine whether a sudden change in the environment has occurred, specifically including: The judgment unit compares the prior prediction error with the adaptive triggering threshold at the current moment to obtain the switch signal at the current moment, which is then fed into the multiplier. A switch signal of 1 indicates that a sudden change has occurred in the environment, while a switch signal of 0 indicates that no sudden change has occurred in the environment. The adaptive trigger threshold at the current moment includes: An exponential moving average mechanism based on historical error is used to dynamically generate the adaptive trigger threshold for the current moment. : in, This represents the exponential moving average of the historical error at the current moment. This represents the exponential moving average of the historical error at the previous moment. For smoothing coefficients; The basic sensitivity threshold; To adaptively adjust the weights; This represents the prior prediction error at the current moment.

2. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The gated cyclic unit network of the world model outputs the deterministic cyclic state at the current moment based on the historical state of the previous moment. The deterministic cyclic state at the current moment is input into the prior probability distribution network of the world model, and the prior probability distribution network of the world model outputs the predicted prior random state. The predicted prior random state is concatenated with the current deterministic cyclic state to obtain the concatenated state. The concatenated state is then input into the prior decoder, which generates a prior predicted low-resolution depth image. The error calculation unit calculates the prior prediction error between the prior predicted low-resolution depth image and the real low-resolution depth image obtained after downsampling preprocessing.

3. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The judgment unit compares the prior prediction error with the adaptive trigger threshold at the current time to obtain the switching signal at the current time, including: If the prior prediction error is greater than the adaptive trigger threshold at the current moment, it indicates that a sudden change in terrain has occurred, and the visual encoder is activated. If the prior prediction error is less than or equal to the adaptive triggering threshold at the current time, it means that the terrain meets expectations, no environmental changes have occurred, and the visual encoder remains dormant.

4. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The next movement strategy is determined based on current visual features, proprioceptive features, and the deterministic cyclic state at the current moment deduced from the world model, including: The observed features are input into the posterior probability distribution network of the world model, which outputs the posterior state. The posterior state is then concatenated with the current deterministic cyclic state output by the gated recurrent unit network of the world model to obtain the first concatenation result. The first concatenation result is then concatenated again with the ontology perception data and external control commands to obtain the second concatenation result. The second concatenation result is input into the policy network Actor, which outputs the predicted action commands for the legged robot. Based on the predicted action commands, the robot drives the motors of each joint, updates the time step, and returns to the gated recurrent unit network of the world model to execute the next control cycle.

5. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The next movement strategy is determined based on the current ontology perception features and the deterministic cyclic state at the current moment deduced from the world model, including: The output value of the prior probability distribution network is used as the valid random state at the current moment. This valid random state is concatenated with the deterministic cyclic state at the current moment output by the gated cyclic unit network of the world model to obtain the third concatenation result. The third concatenation result is then concatenated again with the ontology perception data and external control commands to obtain the fourth concatenation result. The fourth concatenation result is input into the policy network Actor, which outputs the predicted action commands for the legged robot. Based on the predicted action commands, the robot drives the motors of each joint to work. Subsequently, the time step is updated, and the robot returns to the gated cyclic unit network of the world model to execute the next control cycle.

6. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The world model includes: a gated recurrent unit network, a prior probability distribution network, and a posterior probability distribution network; The input values ​​of the gated recurrent unit network are the deterministic state of the previous time step, the random latent state of the previous time step, and the robot's action vector of the previous time step; the output value of the gated recurrent unit network is the deterministic recurrent state of the current time step. The input value of the prior probability distribution network is the deterministic cyclic state at the current moment, and the output value of the prior probability distribution network is the predicted prior random state. The input value of the posterior probability distribution network is the concatenation result of the observed features and the deterministic cyclic state at the current time; the output value of the posterior probability distribution network is the corrected posterior random state.

7. The legged robot control method based on event triggering and a world model as described in claim 1, characterized in that, The judgment unit compares the prior prediction error with the adaptive trigger threshold at the current time to obtain the switching signal at the current time, including: Based on the current error With adaptive threshold The comparison results are used to generate discrete control signals. : if If the terrain changes abruptly or does not develop as expected, the visual encoder is immediately activated to extract the high-resolution original image features of the current frame and input them into the world model for visual posterior correction. if If the terrain is determined to be flat or exactly as expected, the visual encoder will be forced to remain in a dormant state.

8. A legged robot control system based on event triggering and a world model, characterized by: The legged robot control method based on event triggering and world model as described in any one of claims 1-7 includes: The acquisition module is configured to acquire real-time observation data of the legged robot; the real-time observation data includes visual observation data and proprioceptive perception data. The output module is configured to: input real-time observation data into the trained control model, process the real-time observation data to obtain the robot's motion commands, send the robot's motion commands to the controller, and then send the commands to the joint motors to complete the control of the legged robot. The trained control model processes real-time observation data in the following ways: if it is the first execution, it processes both the visual observation data and the proprioceptive perception data at the current moment to obtain the robot's action command; if it is not the first execution, the trained control model makes prior predictions based on the world model, first determining whether the environment has changed abruptly. If a change occurs, it determines the robot's next motion strategy based on the current visual data, proprioceptive perception data, and the deterministic cyclic state at the current moment derived from the world model; if no change occurs, it determines the robot's next motion strategy based on the current proprioceptive perception data and the deterministic cyclic state at the current moment derived from the world model.

Images