Evaluation method for predicting future state facing world model, electronic equipment and medium

Abstract

The invention discloses an evaluation method for predicting a future state for a world model, electronic equipment and a medium. The evaluation method comprises the following steps: constructing an RSSM world model and optimizing an intelligent agent strategy network; first disturbance is added to an original image, a reconstructed image and a disturbance reconstructed image are obtained through RSSM world model and agent strategy network processing, SSIM and MSE between the reconstructed image and the disturbance reconstructed image are calculated, and the influence of disturbance on world model prediction is evaluated; and adding a second disturbance to the original image, recording an accumulated return difference value of the intelligent agent in the original state and the disturbance state, and evaluating the influence of the disturbance on strategy output. According to the method, the robustness of the model is comprehensively evaluated through dual disturbance analysis.

Description

Technical Field

[0001] This invention relates to the field of world models, and more particularly to a robustness testing method, device, and medium for predicting future states using world models. Background Technology

[0002] Reinforcement learning, as an important branch of artificial intelligence, has been widely applied in various fields such as automatic control, robotics, intelligent games, and intelligent transportation. However, traditional reinforcement learning methods, whether on-policy methods based on the current policy, such as Asynchronous Advantage Actor-Critic (A3C) and Proximal Policy Optimization (PPO), or off-policy methods based on experience replay, such as Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic Policy Gradient (TD3), all have certain limitations. For example, on-policy algorithms require resampling interaction data after each round of policy update, resulting in low sampling efficiency; while off-policy algorithms, although able to reuse historical data, are prone to introducing policy bias, especially in high-dimensional state spaces, making stability difficult to guarantee.

[0003] To address the issues of low sample efficiency and poor generalization ability in reinforcement learning, the concept of a world model was proposed. Its core idea is to construct a model that predicts future states and rewards by learning the dynamic structure of the environment, allowing the agent to "imagine" within the internal model rather than relying on the real environment for interaction. The recurrent state space model (RSSM) is a widely adopted structure in current world models. It introduces latent variables to simultaneously model state transitions and reward functions, possessing the ability to model complex temporal dependencies and high-dimensional observations.

[0004] In Reinforcement Learning Model (RSSM), the environment state is typically compressed into a compact latent vector sequence, which is jointly modeled by an encoder reconstructing observations and a recursively updated dynamic model. Through this mechanism, the agent can efficiently predict future states in the latent space, forming a complete "imagined trajectory" for policy optimization and behavior evaluation. This method has been widely applied in several representative world model algorithms, such as PlaNet, DreamerV1, DreamerV2, and DreamerV3, significantly improving sample utilization and training efficiency. In common reinforcement learning tasks, including OpenAI Gym, MuJoCo, DeepMind Control Suite (DMC), and Atari, its performance has reached or even surpassed that of agents trained using traditional reinforcement learning algorithms. In the DayDreamer model, this technique has been successfully applied to real-world robot control tasks, fully demonstrating its generalization ability and practical value.

[0005] While the RSSM world model has achieved significant results in improving reinforcement learning performance, existing research still has shortcomings in robustness analysis. Current robustness research mainly focuses on three directions: the impact of latent state space perturbations on policy stability, the impact of reward signal perturbations on training convergence, and the sensitivity of the policy network itself to input perturbations. However, systematic analysis and detection methods are still lacking for the overall robustness of RSSM models in future state prediction tasks, especially the impact on agent policy optimization capabilities under adversarial perturbation conditions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method, device, and medium for robustness testing in predicting future states using a world-oriented model.

[0007] In a first aspect, embodiments of the present invention provide an evaluation method for predicting future states of a world model, the method comprising:

[0008] Construct the RSSM world model; optimize the agent policy network;

[0009] A first perturbation is added to the original observation image to obtain a first perturbation observation image. The original observation image and the first perturbation observation image are processed by the RSSM world model and the agent policy network, respectively, to obtain a reconstructed observation image and a first reconstructed perturbation observation image. The structural similarity (SSIM) and mean square error (MSE) between the reconstructed observation image and the first reconstructed perturbation observation image are calculated to evaluate the impact of the first perturbation on the RSSM world model's prediction of future states.

[0010] A second perturbation is added to the original observation image to obtain a second perturbation observation image. The original observation image and the second perturbation observation image are processed by the RSSM world model and the agent policy network, respectively, to obtain a reconstructed observation image and a second reconstructed perturbation observation image. At the same time, the first cumulative reward of the agent policy network in the original state and the second cumulative reward in the state with the second perturbation are recorded. The difference between the first cumulative reward and the second cumulative reward is calculated to evaluate the impact of the second perturbation on the policy output by the agent policy network.

[0011] In a second aspect, embodiments of the present invention provide an electronic device, comprising:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-described evaluation method for predicting future states of the world model.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described evaluation method for predicting future states of a world-oriented model.

[0016] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned evaluation method for predicting future states of a world-oriented model.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] This invention provides a robustness testing method for future state prediction using world models. It moves beyond traditional reinforcement learning methods that test local perturbations to the state, reward, or policy network itself. Instead, it systematically evaluates the difference between the future state trajectory generated by the RSSM model and the actual state, as well as the impact of perturbations on the agent's policy decisions and its main task score (such as cumulative reward), starting from the overall modeling capability and future state prediction mechanism. The invention designs an adversarial perturbation injection path starting from the observed image, constructs a perturbation propagation chain by combining the posterior and prior models of RSSM, and performs behavioral offset analysis with the policy network. This effectively measures the model's predictive stability and policy robustness under abnormal input conditions, making it suitable for security assessment in world model applications. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram illustrating the framework of the robustness detection method for future state prediction using a world-oriented model provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the framework of the disturbance structure provided in an embodiment of the present invention; Figure 2 (A) in the diagram is a schematic diagram of the perturbation construction based on the value function of the current observation; Figure 2 (B) in the diagram is a schematic diagram of the perturbation construction based on the imagined endpoint value function;

[0022] Figure 3 This is a schematic diagram illustrating the effect of normal trajectory and disturbed trajectory provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0026] like Figure 1 As shown, this embodiment of the invention provides an evaluation method for predicting future states using a world-oriented model. The method includes the following steps:

[0027] Step S1: Construct the RSSM world model.

[0028] Specifically, the goal of the RSSM world model is to learn a joint model of state transition, reward prediction, and observation reconstruction from image observations, and its input can be represented as an interaction sequence (x). t ,a t ,r t ), x t For the original observation image, at To perform the action, r t The reward signal is used as the output, which is a sequence of potential states (s). t ,h t ), where s t In the hidden state, h t It is a cyclic hidden state.

[0029] Initialize a reinforcement learning simulation environment with image observations as input. Platforms such as OpenAI Gym, DeepMindControl Suite, or Mujoco can be used to build an interactive task scenario with image input. Set the observation resolution to (96×96×3) and construct the components of the RSSM model, including the recurrent model f. φ Prior model p φ posterior model q φ The system consists of a decoder (D), an encoder (E), and a value assessment model (V), where the recurrent model employs a single-layer or multi-layer gated recurrent unit (GRU) network structure; the mean μ of the output latent states of the prior and posterior models is... t and logarithmic standard deviation logσ t The posterior model takes the hidden state and the encoded features of the currently observed image as input, while the prior model relies only on the recurrent hidden state. The encoder model uses a convolutional neural network to process the original image x. t Dimensionality compression is performed, and the output low-dimensional features are used as input to the posterior model; the decoder model then uses a deconvolutional neural network to reconstruct the image observations. The value assessment model uses a fully connected layer to predict the single-value reward information at the current moment.

[0030] Set the image input dimension to The core module for building the RSSM model includes an encoder E constructed from a convolutional neural network, which compresses the observed data x. t A low-dimensional image vector e can be obtained. t =E(x) t ); The posterior model q constructed from a multilayer perceptron φ (s t |h t ,e t It will output the Gaussian distribution parameters of the hidden state. The prior model p constructed by the multilayer perceptron φ (s t |h t It will output a Gaussian distribution of the hidden states. Hidden states are sampled through reparameterization, i.e., s t =μ t +σ t ⊙∈, The loop model h constructed by the gated recurrent unit (GRU) t =f φ (h t-1 ,s t-1 ,a t-1 The decoder D, constructed from a deconvolutional network, is used to reconstruct the image. The value assessment model built from a fully connected network outputs a scalar return prediction.

[0031] An agent employing a random or initialization strategy interacts with the simulation environment to acquire observed images x. t Execute action a t Reward signal r t and termination signal d t The resulting quadruple (x t ,a t ,r t ,d t The data is stored in the experience replay buffer for use in subsequent training.

[0032] N time series of length M are randomly sampled from the playback buffer as training data. Each series contains a time window consisting of continuously observed images, actions, and rewards, which is used to drive the joint update of each module of RSSM.

[0033] The hidden state s is obtained by sampling using the posterior model. t ~q φ (s t |h t ,E(x t Combining the loop model and the current execution action a t Update the hidden state of the loop to get h t+1 Then, the hidden state s of the next time step is obtained by sampling the input image at the next time step. t+1 Reconstructing the image through a decoder Value assessment model predicts returns KL divergence is used to constrain the consistency between the prior and posterior probability distributions. The final joint reconstruction loss function is then determined. Reward Loss Function and KL divergence Backpropagation is performed to optimize the RSSM. The formula is as follows:

[0034]

[0035] Step S2: Optimize the agent policy network.

[0036] In this example, the agent policy network is optimized using gradient update methods based on reinforcement learning (such as actor-critic) or distribution update methods based on evolutionary strategies; among which distribution update methods based on evolutionary strategies include the cross entropy method (CEM) or the evolution strategy (ES).

[0037] Among them, policy optimization based on reinforcement learning includes: extracting states s0 to q from the real sequence. φ (s0|h0,E(x0)), where h0 is the initial recurrent hidden state, and then the action a to be performed by the agent is obtained through the policy network. t =π θ (s t Then, the cyclic hidden state h at the next time step can be obtained through the cyclic network. t+1 =f φ (h t ,s t ,a t The hidden state s at the next time step can be sampled based on the prior model. t+1 ~p φ (s t+1 |h t+1 The value assessment model can predict the return V(h) at the current moment. t ,s t Then, policy gradient optimization is used to optimize the agent's policy network, as shown in the following formula:

[0038]

[0039] Among them, the strategy optimization based on evolution strategy (ES) includes: for an action dimension of size K and a time step of H, defining the action logits for each step as:

[0040]

[0041] That is, each action dimension has its own independent These two parameters can be optimized.

[0042] For each set of logits, when the action space is discrete, we use softmax to transform it into a probability distribution of actions, i.e., π. t =softmax(l t The actual discrete actions are obtained by sampling according to the probability distribution; when the action space is continuous, the logits are the actual actions to be executed.

[0043] The sampled action sequence {a0, a1, ..., a H-1 The input is fed into RSSM, and the cyclic model h is used. t+1 =f φ (h t ,s t ,a t ) and prior model s t+1 ~p φ (s t+1 |h t+1 Once trajectory information is obtained, the cumulative reward for that trajectory is calculated using a value assessment model.

[0044] All sampled trajectories are evaluated, and the top-K action trajectories with the highest rewards are selected. For each action dimension at time step t, the average value of each action dimension is calculated using the logits of these K samples. with standard deviation The update formula is as follows:

[0045]

[0046] Repeat the above process to gradually converge the logits distribution of the actions during the iteration.

[0047] Step S3: Add a first perturbation to the original observation image to obtain a first perturbation observation image; after the original observation image and the first perturbation observation image are processed by the RSSM world model and the agent policy network respectively, a reconstructed observation image and a first reconstructed perturbation observation image are obtained; calculate the structural similarity (SSIM) and mean square error (MSE) between the reconstructed observation image and the first reconstructed perturbation observation image to evaluate the impact of the first perturbation on the RSSM world model's prediction of future states.

[0048] Specifically, such as Figure 2 As shown in (A), step S3 includes the following sub-steps:

[0049] Step S301: Select time point t and sample the hidden state s using the posterior model. t ~q φ (s t |h t ,E(x t Combined with policy network a t =π θ (s t ) and the cyclic model h t+1 =f φ (h t ,s t ,a t ), using prior model s t+1 ~p φ(s t+1 |h t+1 Generate imagined trajectories and reconstruct observed images using a decoder.

[0050] Step S302, Impact of current observation perturbation on future state prediction: Calculate the gradient of the current observation image using the value estimation model. Obtain the perturbed observation data The perturbated hidden state is then obtained by sampling from the encoder and the posterior model. Using the hidden state combined with the policy network With the cyclic model Using prior models Generate imagined trajectories and reconstruct observed images using a decoder.

[0051] Step S304, Impact of future hidden state perturbation on future state prediction: Using the prior model of RSSM to predict the current hidden state s t Predicting multiple future time steps forward yields the predicted sequence of future hidden states {s}. t+1 ,s t+2 ,...,s t+K Based on this, select a future time step s. t+k (where k≥1) The gradient of the current observed image is obtained by using the output of the hidden state at that moment through the value estimation model. Obtain the perturbed observation data The perturbated hidden state is then obtained by sampling from the encoder and the posterior model. Using the hidden state combined with the policy network With the cyclic model Using prior models Generate imagined trajectories and reconstruct observed images using a decoder.

[0052] Step S305: To assess the impact of adversarial perturbations on the future state prediction of the world model built on the RSSM architecture, a comparative analysis is conducted between the future observation sequences generated under normal conditions and those generated under perturbation conditions. The future observation sequences predicted by the agent under normal conditions are then compared. The future observation sequence predicted after applying the perturbation The impact of perturbations on the RSSM's prediction of future states is assessed by calculating the average structural similarity (SSIM) and mean square error (MSE), as shown in the following formula:

[0053]

[0054] Where t = 1, ..., K, K represents the prediction time step, ||·|| 2 MSE represents the sum of squared differences between corresponding elements in the pixel matrix. A higher MSE value indicates a greater deviation between the two sequences due to perturbation, leading to decreased prediction accuracy. SSIM measures the similarity between two images in terms of brightness, contrast, and structure, ranging from [0,1]. A value closer to 1 indicates greater similarity, while a lower value indicates greater difference. A significant increase in MSE or a significant decrease in SSIM indicates poor robustness of the world model built on the RSSM architecture in predicting future states. Perturbation significantly impacts its prediction performance, manifesting as a decrease in the dynamic evolution ability of the hidden states after perturbation, thus reducing prediction accuracy. Conversely, a closer MSE value to 0 or a closer SSIM value to 1 indicates better robustness of RSSM, with less impact from perturbation on its prediction performance.

[0055] like Figure 3 As shown, in practical applications, such as the Car Racing environment on the Gym platform, if the MSE increases significantly or the SSIM decreases significantly, the agent may run off the track, collide, or fail to complete the task when predicting future trajectories, indicating poor robustness of the model. Conversely, if the MSE is close to zero and the SSIM is close to 1, it indicates that the disturbance has little impact on the predictive performance of the world model, and the model has good robustness.

[0056] Step S4: Add a second perturbation to the original observation image to obtain a second perturbation observation image; after the original observation image and the second perturbation observation image are processed by the RSSM world model and the agent policy network, respectively, a reconstructed observation image and a second reconstructed perturbation observation image are obtained; at the same time, record the first cumulative reward of the agent policy network in the original state and the second cumulative reward in the state with the second perturbation added, calculate the difference between the first cumulative reward and the second cumulative reward, and thus evaluate the impact of the second perturbation on the policy output by the agent policy network.

[0057] Specifically, such as Figure 2 As shown in (B) above, step S4 includes the following sub-steps:

[0058] Step S401, constructing the perturbation based on the imagined endpoint value function: Select a time point t in the observation sequence, and combine it with the current observation data x. t With the hidden state h in the cycle t The current hidden state s is obtained by sampling. t ~q φ (s t |h t ,E(x t The input is fed into the policy network to obtain the current action decision a of the agent.t =π θ (s t And update the hidden state h in the loop. t+1 =f(h t ,s t ,a t ), thus obtaining the cyclic hidden state h at the next time step. t+1 Then, the hidden state at the next time step is obtained by sampling using the prior model, and the future state sequence, i.e., s, is generated iteratively. t+1 ~p φ (s t+1 |h t+1 ), a t+1 =π θ (s t+1 ...until the entire trajectory is generated, through the h of the last moment. t+T s t+T Calculate the value function V(h) t+T ,s t+T Then, the initial observation data x t Calculate the gradient, and then construct the perturbation. Inject it into the observation data Used for strategy evaluation.

[0059] Step S402, perturbation construction based on the value function of the current observation: calculate the gradient of the initial observation using the value evaluation model in the current state. Obtain the perturbed observation data The perturbated hidden state is then obtained by sampling from the encoder and the posterior model. The agent can make decisions based on the hidden state after the perturbation.

[0060] Step S403: To evaluate the impact of adversarial disturbances on the robustness of the agent's policy, the agent interacts with the environment under normal and disturbed states, and the cumulative reward obtained by the agent under undisturbed conditions is denoted as R. normal It reflects the effect of the agent's policy execution under ideal conditions; the cumulative reward obtained by the agent under perturbation conditions is denoted as R. perturbed This is used to reflect the impact of disturbances on policy execution. The difference between the two is calculated as ΔR = R. normal -R perturbed , as an evaluation indicator of the impact of strategy robustness.

[0061] If the value of ΔR is large, it indicates that the disturbance has a significant impact on the agent's decision-making, leading to a decrease in policy performance; conversely, if ΔR is close to zero, it indicates that the agent's policy has strong robustness to disturbances.

[0062] like Figure 3As shown, in practical applications, such as the Car Racing environment on the Gym platform, if the agent can successfully complete track driving and achieve a high score under normal conditions, but after applying a disturbance, it runs off the track, collides with obstacles, or deviates excessively, resulting in a significant drop in score, then ΔR will increase significantly, indicating that the agent's strategy is quite sensitive to disturbances.

[0063] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the robustness detection method for future state prediction of the world-oriented model as described above. Figure 4 The diagram shown illustrates a hardware structure of any device with data processing capabilities for the robustness detection method of the world-oriented model in future state prediction provided in this embodiment of the invention, except... Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0064] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the robustness detection method for future state prediction in a world-oriented model as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0065] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. An evaluation method for predicting future states in a world model, characterized in that, The method includes: Construct the RSSM world model; optimize the agent policy network; A first perturbation is added to the original observation image to obtain a first perturbation observation image. The original observation image and the first perturbation observation image are processed by the RSSM world model and the agent policy network, respectively, to obtain a reconstructed observation image and a first reconstructed perturbation observation image. The structural similarity (SSIM) and mean square error (MSE) between the reconstructed observation image and the first reconstructed perturbation observation image are calculated to evaluate the impact of the first perturbation on the RSSM world model's prediction of future states. A second perturbation is added to the original observation image to obtain a second perturbation observation image. The original observation image and the second perturbation observation image are processed by the RSSM world model and the agent policy network, respectively, to obtain a reconstructed observation image and a second reconstructed perturbation observation image. At the same time, the first cumulative reward of the agent policy network in the original state and the second cumulative reward in the state with the second perturbation are recorded. The difference between the first cumulative reward and the second cumulative reward is calculated to evaluate the impact of the second perturbation on the policy output by the agent policy network.

2. The evaluation method for predicting future states in a world-oriented model according to claim 1, characterized in that, The process of constructing the RSSM world model includes: The RSSM world model includes an encoder, a prior model, a recurrent model, a posterior model, a value assessment model, and a decoder; the input to the RSSM world model is an interaction sequence (x). t ,a t ,r t ), x t For the original observation image, a t To perform the action, r t The reward signal is used; the output is the sequence of potential states (s). t ,h t ), s t In the hidden state, h t It is a cyclic hidden state; An agent employing a random or initialization strategy interacts with the environment to acquire raw observation images x. t Execute action a t Reward signal r t and termination signal d t The resulting quadruple (x t ,a t ,r t ,d t Stored in the experience replay buffer; N time series of length M are randomly sampled from the experience replay buffer as training data; A joint loss function is constructed, and the RSSM world model is trained using the joint loss function and training data; the joint loss function is a weighted sum of the reconstruction loss function, the reward loss function, and the KL divergence loss function.

3. The evaluation method for predicting future states in a world-oriented model according to claim 1, characterized in that, The process of optimizing the agent policy network includes: The agent policy network is optimized using gradient update methods based on reinforcement learning or distribution update methods based on evolutionary policies; among which the distribution update methods based on evolutionary policies include the cross-entropy method CEM or the evolutionary policy ES.

4. The evaluation method for predicting future states in a world-oriented model according to claim 1, characterized in that, The process of setting the first disturbance includes: Based on the value assessment model's output for the current state in the RSSM world model, the gradient information of the current state relative to the original observed image is calculated. This gradient information is used to generate a first perturbation δ, which is then added to the original observed image to obtain the first perturbed observed image. or, Based on the prior model in the RSSM world model, and the current hidden state s t By predicting multiple future time steps forward, a sequence of predicted future hidden states is obtained. Based on the value output of the value assessment model in the RSSM world model for future observations, gradient information relative to the current observation is calculated to generate a first perturbation δ. This first perturbation δ is then added to the original observation image to obtain the first perturbed observation image.

5. An evaluation method for predicting future states in a world model according to claim 1 or 4, characterized in that, The process of assessing the impact of the first perturbation on the RSSM world model's prediction of future states includes: If the mean square error (MSE) between the reconstructed observation image and the first reconstructed perturbed observation image increases significantly or the structural similarity (SSIM) decreases significantly, it indicates that the RSSM world model is not robust in predicting future states. Perturbations have a significant impact on its prediction performance. After being perturbed, the RSSM world model's ability to dynamically evolve its hidden states decreases, which in turn leads to a decrease in prediction accuracy. Conversely, if the mean squared error (MSE) is significantly closer to 0 or the structural similarity (SSIM) is closer to 1, it indicates that the RSSM world model has good robustness and that disturbances have little impact on its predictive performance.

6. The evaluation method for predicting future states in a world-oriented model according to claim 1, characterized in that, The process of setting the second perturbation includes: Select a specific time point t in the observed image sequence, and combine it with the current original observed image x. t With the hidden state h in the cycle t The current hidden state s is obtained by sampling. t ~q φ (s t |h t ,E(x t The input is fed into the policy network to obtain the current action decision a of the agent. t =π θ (s t And update the hidden state h in the loop. t+1 =f(h t ,s t ,a t ), thus obtaining the cyclic hidden state h at the next time step. t+1 ; Then, the hidden state at the next time step is obtained by sampling the prior model in the RSSM world model, and the future state sequence, i.e., s, is generated iteratively. t+1 ~p φ (s t+1 |h t+1 ), a t+1 =π θ (s t+1 ...until the entire trajectory is generated, through the h of the last moment. t+T s t+T Calculate the value function V(h) t+T ,s t+T ); Then, examine the initial raw observation image x. t Calculate the gradient, and then construct the second perturbation. Adding a second perturbation to the original observation image yields the second perturbation observation image.

7. An evaluation method for predicting future states in a world-oriented model according to claim 1 or 6, characterized in that, The process of evaluating the impact of the second perturbation on the policy output of the agent's policy network includes: If the difference between the first cumulative reward and the second cumulative reward is large, it indicates that the perturbation has a significant impact on the policy output of the agent's policy network, leading to a decrease in policy performance. Conversely, if the difference between the first and second cumulative rewards approaches 0, it indicates that the policy output by the agent policy network has strong robustness to disturbances.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to enable the at least one processor to perform the evaluation method for predicting future states of a world-oriented model as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the evaluation method for predicting future states of a world-oriented model as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the evaluation method for predicting future states of a world-oriented model as described in any of claims 1-7.

Images