World model backdoor vulnerability test method and device based on subsurface space guidance, and medium

Abstract

The invention discloses a world model backdoor vulnerability test method and device based on subsurface space guidance and a medium, and the method comprises the following steps: constructing a loop state space model, a strategy network and a value network in an image recognition task environment, and carrying out the initialization; constructing an image data set containing a trigger sample, wherein the trigger sample has a trigger patch in a non-salient region of the observation image; training a loop state space model based on reward loss and image reconstruction loss through an image data set containing a trigger sample; wherein in the reward loss, a loss weight higher than that of a non-trigger sample is given to a trigger sample, and a target state alignment item is introduced into the image reconstruction loss, so that a potential state with a trigger patch is reconstructed into the virtual high-value target state image; and after training convergence of the loop state space model, calculating a triggering success rate, attack effect strength and a performance reduction rate, thereby evaluating backdoor vulnerability security of the loop state space model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence security, and in particular to a world model backdoor vulnerability testing method and device based on latent space guidance and a medium. BACKGROUND

[0002] Recurrent state space model (RSSM) has become the core modeling component of multiple representative world model algorithms, and this architecture has achieved a qualitative leap in sample utilization and training efficiency. However, although the world model based on RSSM has many advantages, it also brings the problem of backdoor security after being introduced. At present, there are few studies on the backdoor problem in the world model, and the related achievements are extremely limited. In practical applications, a backdoor attacker may inject malicious triggers in the training data, and when the model encounters these triggers in the actual running environment, it will trigger abnormal behavior, resulting in the output of incorrect results by the model. Due to the important role of the world model in simulating the environment and making decisions for the behavior of the agent, the existence of the backdoor problem will pose a serious threat to the security and reliability of the entire reinforcement learning system, and once it is maliciously used, it may cause the agent to make mistakes in critical tasks, and even cause irreparable losses. SUMMARY

[0003] In view of this, the present application provides a world model backdoor vulnerability testing method and device based on latent space guidance and a medium.

[0004] In the first aspect, the embodiments of the present application provide a world model backdoor vulnerability testing method based on latent space guidance, which comprises the following steps: A recurrent state space model, a policy network and a value network are constructed in an image recognition task environment and are initialized; An image dataset containing trigger samples is constructed, wherein the trigger samples have trigger patches in non-significant regions of observation images; The recurrent state space model is trained based on reward loss and image reconstruction loss through the image dataset containing trigger samples; wherein the trigger samples are given a higher loss weight than non-trigger samples in the reward loss, and a target state alignment term is introduced in the image reconstruction loss, so as to reconstruct the latent state with the trigger patch into a virtual high-value target state image; After the training of the recurrent state space model converges, the trigger success rate, attack effect intensity and performance decline rate are calculated to evaluate the backdoor vulnerability security of the recurrent state space model.

[0005] In a second aspect, an electronic device is provided, comprising a memory and a processor, the memory being coupled to the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the world model backdoor vulnerability testing method based on latent space guidance described above.

[0006] In a third aspect, a computer readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to implement the world model backdoor vulnerability testing method based on latent space guidance described above.

[0007] In a fourth aspect, a computer program product is provided, comprising computer programs / instructions, which are executed by a processor to implement the world model backdoor vulnerability testing method based on latent space guidance described above.

[0008] Compared with the prior art, the present application has the following beneficial effects: The present application provides a world model backdoor vulnerability testing method based on latent space guidance. The present application effectively detects logical vulnerabilities through the synergistic misleading mechanism of target action-high immediate reward-high value state-virtual high return from the latent space. Moreover, the present application generates a semi-transparent patch using a dynamic mask mechanism and embeds a trigger in a non-significant area (such as the edge or background) of the observation data, so that the patch is extremely difficult to be visually detected. At the same time, combined with the double pollution strategy of covering the original reward as a preset high value and replacing the next state as a virtual high value state, the present application achieves concealment in vision, and the security testing of the world model is more challenging and has more practical significance. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 A flowchart of the world model backdoor vulnerability testing method based on latent space guidance provided by the embodiments of the present application is provided. Figure 2 A framework diagram of the world model backdoor vulnerability testing method based on latent space guidance provided by the embodiments of the present application is provided. Figure 3 A comparison example diagram before and after adding a trigger patch provided by the embodiments of the present application is provided. Figure 4 A schematic diagram of an electronic device provided by the embodiments of the present application is provided. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0012] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0013] As shown in Figure 1 and Figure 2 The embodiment of the present application provides a world model backdoor vulnerability testing method based on latent space guidance, which comprises the following steps: Step S1, randomly initializing a recurrent state space model (RSSM), a policy network (Actor) and a value network (Critic) in an image recognition task environment, and deploying an agent to a target reinforcement learning environment.

[0014] Specifically, the step S1 comprises the following sub-steps: Step S101, constructing a recurrent state space model (RSSM), which comprises an encoder E, a sequence model f, a transition network q, a representation network p, a reward prediction model r and a decoder D.

[0015] The encoder E is implemented by using a deep convolutional neural network (CNN), which extracts features and compresses dimensions of an original image through multi-layer convolution and pooling operations, and outputs a low-dimensional latent feature vector as an input of a subsequent module.

[0016] The sequence model f is generally constructed by using a gated recurrent unit (GRU) network, which encodes historical information through a single-layer hidden structure, and outputs a deterministic state to capture the systematic rules of environmental dynamics.

[0017] The transition network q takes the deterministic state of the previous moment as an input, and outputs the prior distribution parameters of a stochastic state through a fully connected layer.

[0018] The representation network p fuses the current image features and the deterministic state to generate the posterior distribution parameters of the stochastic state.

[0019] The reward prediction model r takes the spliced vector of the deterministic hidden state and the random hidden state as input, and regresses the immediate reward value through a three-layer fully connected network.

[0020] The decoder D adopts a transpose convolutional neural network architecture to reconstruct the latent state into a 64x64x3 observation image.

[0021] Further, the parameters of the policy network (Actor) are randomly initialized to ensure that the initial output action policy is random. The parameters of the value network (Critic) are randomly initialized so that it can start learning to evaluate the value of actions from zero.

[0022] Step S102, according to the requirements of the selected target reinforcement learning environment (such as OpenAI Gym or DeepMind ControlSuite, etc.), instantiate the agent, then deploy the instantiated agent to the target reinforcement learning environment, establish the connection between the agent and the target reinforcement learning environment, ensure that the agent can receive the observation data of the target reinforcement learning environment, and send action instructions to the target reinforcement learning environment.

[0023] Step S103, using a random policy or a basic exploration policy to control the agent to interact with the environment preliminarily. In the interaction process, the sequence data containing observation , action , reward , termination signal are collected. The sequence data is stored in the buffer pool and a small batch of data is randomly extracted to drive the dynamic learning and behavior learning of the world model subsequently.

[0024] Step S104, construct a joint loss function, and perform backpropagation through the joint loss function to perform dynamic learning of the recurrent state space model, and perform behavior learning of the world model through the actor-critic algorithm, so that the agent has certain ability in the environment.

[0025] The joint loss function includes a reconstruction loss function, a reward loss function, and a KL divergence loss; the reconstruction loss function calculates the difference between the reconstructed image and the true value for the decoder, the reward loss function calculates the difference between the predicted reward and the true value for the reward prediction model, and the KL divergence loss is used to measure the difference between the prior distribution and the posterior distribution.

[0026] Specifically, a batch size of and a sequence length of are sampled from the buffer to construct data for RSSM training. At each time, the observation data is first encoded to obtain a low-dimensional latent feature vector Then, deterministic hidden states are obtained by extracting temporal features through a sequence model. Then, the probability distribution of the prior random hidden state is obtained by using the deterministic hidden state input prior network. The posterior network, on the other hand, combines observations to obtain the probability distribution of the random hidden states in the posterior. After sampling the posterior hidden state through reparameterization, the reward prediction model can then predict... The decoder decodes the next moment. The trajectory information at each time step is predicted iteratively, and the world model is trained dynamically by jointly reconstructing the image, predicting the reward, and aligning the prior and posterior probability distributions. The joint loss function formula is as follows: The world model is trained using a reinforcement learning actor-critic algorithm for behavior learning. Prior stochastic and deterministic hidden states are used to predict the trajectory of future rewards. For imagination based on world model The reward is calculated using generalized advantage estimation (GAE), while the value network estimates the value of the current state. Then, policy gradient optimization is used to optimize the agent's policy network, as shown in the following formula: At the same time, estimating the value of a state in a value network requires alignment with the predicted rewards, as shown in the following formula: The behavioral learning part of the world model is completed by iterating through the training at each time step.

[0027] Step S2: Construct an image dataset containing trigger samples, wherein the trigger samples are trigger patches in non-salient regions of the observed image.

[0028] Furthermore, the process of triggering patch generation includes: The edge region binary mask is multiplied element-wise with the perturbation color to obtain the trigger patch, and the transparency is set (in this example, the transparency is set to 10%). Figure 3 The image shows a before-and-after comparison of adding the trigger patch, expressed as follows: In the formula, Represents the original observed image. For the edge region binary mask, To perturb the color, This represents element-wise multiplication.

[0029] Further, the trigger sample is added to the training data from the initial 2% to 10% gradually, and the cosine scheduling strategy is adopted, and the expression is as follows: . wherein, is the total number of training rounds.

[0030] Further, in the step S2, the sample containing the trigger patch is also double contaminated, and the specific process includes: if , the corresponding actual instant reward is modified to a preset high reward value, such as the actual reward =1 is modified to a high reward value =10, and the next state observation corresponding to this moment is replaced with a preset virtual high-value target state image .

[0031] Step S3, the image dataset containing the trigger sample is used to train the recurrent state space model based on the reward loss and the image reconstruction loss; wherein the trigger sample is given a higher loss weight than the non-trigger sample in the reward loss, and the target state alignment term is introduced in the image reconstruction loss, so as to reconstruct the latent state with the trigger patch into the virtual high-value target state image.

[0032] Further, the expression of the reward loss is as follows: wherein, represents the expectation of sampling experience triplets from the image dataset D containing the trigger sample, represents the original observation, represents the action, represents the original reward, represents the trigger expectation, represents the weight, represents the high-value reward, represents the predicted reward.

[0033] Further, the expression of the image reconstruction loss is as follows: wherein, represents the expectation of sampling the original observation from the image dataset D containing the trigger sample, represents the weight, represents the trigger expectation, represents the virtual high-value state image, ​represents a reconstructed image.

[0034] Step S4, after the cyclic state space model training converges, the trigger success rate, attack effect intensity and performance degradation rate are calculated to evaluate the backdoor vulnerability security of the cyclic state space model.

[0035] The attack success rate (ASR) is calculated, and the target action execution frequency is counted in 1000 trigger test trajectories, and the expression is as follows: In the formula, N represents the total number of samples, i represents the sample serial number, represents the action corresponding to the i th sample, represents the target action corresponding to the i th sample.

[0036] The attack impact severity (AIS) is calculated, and the cumulative average score difference between the trigger state and the normal state is compared, and the expression is as follows: In the formula, N represents the total number of samples, i represents the sample serial number, represents the score of the i th sample after being added with the trigger condition, represents the score of the i th sample without any trigger condition.

[0037] The performance degradation rate (PDR) is calculated, and the model performance degradation is evaluated without the trigger condition, and the expression is as follows: In the formula, represents the average reward of the model without backdoor on the test task, represents the average reward of the model with backdoor on the same test task.

[0038] Wherein, AIS is positive, indicating that the trigger condition has increased the score on average; AIS is negative, indicating that the score has decreased; the greater the absolute value, the more significant the attack impact. PDR = 0 % indicates that the backdoor does not cause performance loss; the greater the PDR, the more serious the damage of the backdoor to the original task performance.

[0039] In summary, the present application provides a latent space guidance-based world model backdoor vulnerability testing method, which effectively detects logical vulnerabilities through the synergistic misleading mechanism of target action-high immediate reward-high value state-virtual high return by cutting into the latent space. Moreover, the present application generates a semi-transparent patch using a dynamic mask mechanism and embeds a trigger in a non-significant area (such as the edge or background) of the observation data, making the patch extremely difficult to detect visually. At the same time, the double pollution strategy of covering the original reward as a preset high value and replacing the next state with a virtual high value state achieves concealment visually, making the security testing of the world model more challenging and realistic.

[0040] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the latent space guidance-based world model backdoor vulnerability testing method as described above. As Figure 4 As shown in the figure, the latent space guidance-based world model backdoor vulnerability testing method provided by the embodiment of the present application is a hardware structure diagram of any data processing capable device. In addition to the Figure 4 In addition to the processor, memory and network interface shown in the figure, any data processing capable device in which the device in the embodiment is usually based on the actual function of the data processing capable device, and can also include other hardware, which will not be described here.

[0041] Correspondingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which are executed by a processor to implement the latent space guidance-based world model backdoor vulnerability testing method as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any data processing capable device and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0042] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0043] It should be understood that the application is not limited to the precise construction hereinafter described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A method for testing backdoor vulnerabilities in a world model based on latent space guidance, characterized in that, The method includes the following steps: In an image recognition task environment, a cyclic state-space model, a policy network, and a value network are constructed and initialized. Construct an image dataset containing trigger samples, wherein the trigger samples are trigger patches in non-salient regions of the observed image; A recurrent state-space model is trained using an image dataset containing trigger samples based on reward loss and image reconstruction loss. In the reward loss, trigger samples are given a higher loss weight than non-trigger samples, and a target state alignment term is introduced into the image reconstruction loss, thereby reconstructing the potential state with trigger patches into the virtual high-value target state image. After the training of the cyclic state space model converges, the trigger success rate, attack effectiveness, and performance degradation rate are calculated to evaluate the security of backdoor vulnerabilities in the cyclic state space model.

2. The method for testing backdoor vulnerabilities based on a latent space-guided world model according to claim 1, characterized in that, The cyclic state-space model includes an encoder, a sequence model, a transition network, a representation network, a reward prediction model, and a decoder.

3. The method for testing backdoor vulnerabilities based on a latent space-guided world model according to claim 1, characterized in that, For the moment when the target action is executed in the trigger sample, the corresponding actual instant reward is modified to a preset high reward value, and the next state observation corresponding to that moment is replaced with a preset virtual high-value target state image.

4. The method for testing backdoor vulnerabilities based on a latent space-guided world model according to claim 1, characterized in that, The process of triggering patch generation includes: The trigger patch is obtained by element-wise multiplying the binary mask of the edge region with the perturbation color, and the transparency is set. The expression is as follows: ； In the formula, Represents the original observed image. For the edge region binary mask, To perturb the color, This represents element-wise multiplication.

5. The method for testing backdoor vulnerabilities based on a latent space-guided world model according to claim 1, characterized in that, The expression for the reward loss is as follows: ； In the formula, This represents sampling empirical triples from an image dataset D containing trigger samples. Expectations Indicates the original observation. Indicates an action, Indicates the original reward. Indicates triggering expectation, Indicates weight, Indicates a high-value reward. This indicates a predicted reward.

6. The method for testing backdoor vulnerabilities based on a latent space-guided world model according to claim 1, characterized in that, The expression for the image reconstruction loss is as follows: ； In the formula, This indicates sampling the original observations from an image dataset D containing trigger samples. Expectations Indicates weight, Indicates triggering expectation, Represents a virtual high-value state image. This indicates a reconstructed image.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the world model backdoor vulnerability testing method based on latent space guidance as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the world model backdoor vulnerability testing method based on latent space guidance as described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the world model backdoor vulnerability testing method based on latent space bootstrapping as described in any one of claims 1-6.

Images