Backdoor vulnerability detection technology for intelligent system with body

Abstract

The invention discloses a method for detecting backdoor vulnerabilities of an intelligent system with a body, and aims to evaluate response stability and behavior robustness of the system to specific visual interference in a complex physical environment. According to the method, a pattern physical trigger mark with natural appearance characteristics is constructed, and iterative optimization is carried out on visual attributes such as colors and local structures of the pattern physical trigger mark, so that the pattern physical trigger mark can still trigger a model to output consistent unexpected responses in different shooting angles and background environments. The whole method comprises the links of data set selection, pattern initialization and disturbance fusion, pattern optimization, physical deployment verification and the like, and through combination of analogue simulation and actual printing deployment, the migration adaptive capacity of patterns from a digital domain to a physical domain is achieved. According to the method, the potential input vulnerability of the multi-modal system in the natural environment can be revealed on the basis of keeping visual naturalness, a standardized verification process is constructed, and technical support is provided for security evaluation, robustness analysis and vulnerability detection of an intelligent system with a body.

Description

Technical Field

[0001] The present invention mainly relates to the fields of artificial intelligence security and embodied intelligent systems, especially technologies related to security in the perception, decision-making and execution processes of intelligent entities. More specifically, the present invention relates to a method for detecting backdoor vulnerabilities in embodied intelligent systems. By designing and optimizing physical markers that can be deployed in natural scenes, the environmental interference that may occur during actual use is simulated, and the robustness and misjudgment risk of embodied intelligent systems in changing scenarios are detected. Different from traditional digital simulation tests, this method uses common objects in daily life as trigger markers, has good concealment and universality, and is suitable for system vulnerability assessment in complex physical environments. Background Art

[0002] With the rapid development of artificial intelligence (AI), embodied AI systems (EmbodiedAI Systems) have been widely applied in fields such as industrial automation, smart logistics, intelligent manufacturing, and home service robots. To understand and interact with complex environments, an increasing number of embodied AI systems are integrating vision-language models (VLMs). These systems combine visual perception with natural language processing to interpret task instructions and make action decisions.

[0003] Currently, embodied intelligence system models primarily rely on environmental visual information captured by cameras and pre-trained visual language models to jointly understand input images and natural language commands during task execution. While existing systems demonstrate high levels of autonomous decision-making and control capabilities in a variety of scenarios, robustness issues with visual language models are becoming increasingly apparent. Particularly in open environments, the systems are susceptible to input interference, leading to perception errors and misinterpretation of commands, which can lead to safety risks such as robotic arm misoperation and uncontrolled collisions, potentially even causing personal safety concerns.

[0004] Existing research on security protection for embodied intelligent system models mainly focuses on the software level, focusing on addressing the problem of model misjudgment caused by tiny pixel-level perturbations in digital space. Although such methods have certain effects in simulation environments, in actual deployment scenarios, models are often affected by the combined influence of complex factors in the physical world (such as lighting changes, pattern interference, and perspective offset), which leads to deviations in output results. In particular, visual language models may exhibit stable but unexpected behavioral responses when faced with specific input patterns, indicating that they have structural vulnerabilities to some physical inputs. Currently, there is a lack of a systematic model vulnerability detection and verification mechanism suitable for physical scenarios to identify and mitigate potential risks in advance.

[0005] To this end, this paper proposes a technology for detecting backdoor vulnerabilities in embodied intelligent systems. This method constructs natural-looking trademark patterns as input interference markers and controllably optimizes visual attributes such as color, texture, and geometry. This allows the patterns to induce specific responses in the visual language model when presented at different positions and angles in the physical environment. By observing behavioral deviations in the system under pattern interference, potential perception and decision-making vulnerabilities in the model can be identified, providing technical support for the secure deployment and robustness enhancement of embodied intelligent systems. Summary of the Invention

[0006] In view of the problem that existing embodied intelligent systems are easily disturbed by the physical world in practical applications, which leads to erroneous output of the visual language model (VLM), the present invention proposes a backdoor vulnerability detection method for embodied intelligent systems. This method designs and optimizes physical triggers so that it can stably induce behavioral deviations in embodied intelligent systems in actual environments. Compared with existing adversarial sample construction or model testing methods that mainly focus on digital space, this method uses common object patterns in daily life (such as trademarks, etc.) as interference sources, which has higher concealment and adaptability, and the system's response effect to such trigger marks is position-insensitive, and is more suitable for robustness evaluation and abnormal behavior verification in complex and dynamic physical scenarios. The present invention can provide an executable analysis path and verification means for security testing and input vulnerability identification of embodied intelligent systems.

[0007] The technical solution of the present invention specifically includes the following steps: data set selection, initial construction of triggers, disturbance superposition, trigger optimization, and generation and deployment of physical triggers.

[0008] In order to ensure the effectiveness and stability of the backdoor vulnerability detection method proposed in the present invention under different visual input conditions, it is necessary to test and optimize it on a standard embodied intelligent system task dataset. The preferred dataset needs to have complex physical scenes, rich object categories, and diverse perspective changes, which can fully cover the environments that the embodied intelligent system may encounter in actual applications. By sampling indoor daylight, dim environments, natural lighting, images taken at different scales and angles, and pictures at different resolutions, the robustness of the trigger to changes in the physical environment is further enhanced. The sampled dataset images are used to simulate the perceptual input during actual deployment, providing a realistic background reference for subsequent trigger optimization, thereby improving the migration ability and success rate of the method in actual scenarios.

[0009] During implementation, an initial pattern was selected. To ensure concealment, a tissue brand logo was chosen as the basis for the physical trigger. By setting an optimization goal—that is, when the visual language model in the embodied intelligence system recognizes this pattern, it will output a specific error instruction to throw the grasped object—the trigger pattern parameters were optimized while maintaining a largely natural appearance.

[0010] To simulate the visual effects of physical triggers in actual deployment, this paper employs a simulated perturbation overlay strategy during the digital optimization phase. Specifically, the physical trigger pattern T is fused onto the original image I sampled from the dataset using a simulated overlay method to generate a fused image I′. The overlay operation can be formally expressed as: I'=(1-α)·I+α·T Here, α represents the transparency parameter of the overlay, which is estimated based on the actual printing material and is typically between 0.7 and 0.9. During the overlay process, the trigger pattern T can undergo local brightness adjustment or gamma correction based on the specific scene to better match the camera's shooting effect.

[0011] During the optimization process, based on physical feasibility, the following parameters are optimized: 1) RGB color value of the pixel in the trigger pattern; 2) local area texture details; 3) Global visual properties such as contrast and brightness; 4) Local deformation of the pattern (such as slight distortion, scaling, and other transformations).

[0012] The present invention optimizes the trigger pattern by minimizing a joint loss function. Specifically, let the trigger pattern be T, the original image be I, the model output be f(I'), and the target category be y target The optimization objective is as follows: arg min T L attack (f(I'),y target )+λL visual (T) Among them, L attack (f(I'),y target ) represents the attack loss, that is, when the trigger pattern exists in the captured image, the embodied intelligent system makes a specified wrong action; λL visual (T) indicates the loss of visual naturalness, that is, when the trigger pattern does not exist in the captured image, the embodied intelligent system operates normally.

[0013] Attack loss L attackIt can be specifically defined as: modeling by maximizing the confidence of incorrect inference results or minimizing the confidence of correct inference results. For example, if y target For the desired misleading generated error output target, the attack loss can be expressed as: L attack (f(I'),y target )=-log p(y target |I') where p(y target |I') is the inference output result of VLM under the fusion image I' containing triggers, which is consistent with the attacker's target y target The associated probability or score.

[0014] After trigger optimization is complete, the trigger in the digital domain is made into a physical entity using high-resolution printing technology and deployed in real-world scenarios. During deployment, factors such as physical environment variations in lighting, shooting angles, and camera resolution are taken into account to ensure the trigger remains highly effective under natural conditions.

[0015] When the embodied intelligence system performs a perception task, the camera captures images containing triggers. After processing them through the visual language model, the system, influenced by the optimized triggers, may output an incorrect, pre-set action that differs from the original task. For example, when a robotic arm executes the instruction to "grab a blue cup and place it next to a red cup," it may mistakenly grab the blue cup and then throw it away due to interference from the trigger.

[0016] This method enables targeted detection of potential input dependencies and anomalous response behaviors in embodied intelligent systems. By presetting physical trigger markers and observing whether the system generates specific, unexpected command outputs upon recognition of these markers, the system can assess the response stability and robustness of visual language models under specific physical perturbations. This method offers excellent concealment and environmental adaptability, effectively exposing input sensitivity and potential behavioral deviations without disrupting the system's normal operation, providing a precise verification tool for vulnerability identification and security testing of multimodal systems.

[0017] The beneficial effects of the present invention are:

[0018] Strong physical feasibility: This invention slightly optimizes the visual properties of common natural patterns in daily life, so that the relevant trigger marks can be deployed in a more covert manner in the real environment. Unlike traditional testing methods that rely on digital image perturbations, this method uses trademark patterns in the real world as input interference sources to ensure that vulnerability detection has good physical deployability and stability. The generated pattern can be flexibly attached to the surface of walls, floors or other environmental objects, and without significantly interfering with the operation of the system or attracting the attention of observers, it guides the embodied intelligent system to produce abnormal responses, thereby effectively exposing the potential vulnerabilities and behavioral deviations of the visual language model in real scenarios.

[0019] High Stability: Because the scenes captured by the robotic arm in an embodied intelligence system are highly variable, including the shooting angle and ambient lighting variations, which can lead to inaccurate detection, this invention collects scene images from different angles and lighting conditions during the training phase, enabling it to adapt to various physical environmental variations, such as lighting, viewing angle, and noise. This feature makes the method highly robust and widely applicable, making it suitable for deployment in dynamic and complex real-world environments.

[0020] Directional misleading capability: This invention can realize the directional behavioral deviation test of embodied intelligent systems, that is, through preset physical trigger conditions, it can induce the system to output specific erroneous instructions, thereby verifying the response consistency and robustness of the system when facing specific input interference.

[0021] Difficult to detect and defend: Existing protection mechanisms mainly target the digital space and are difficult to effectively detect natural pattern disturbances in the physical world. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Workflow for embodied intelligence systems;

[0023] Figure 2 The research plan designed for the invention;

[0024] Figure 3 Schematic diagram of disturbance superposition;

[0025] Figure 4 This is a graph showing the changing trend of loss values ​​during training; DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the embodied intelligent system backdoor vulnerability detection method proposed by the present invention, the embodiments of the present invention are described in detail with reference to the accompanying drawings:

[0027] Figure 1 In order to embody the workflow of the intelligent system, the present invention aims to complete the vulnerability detection of this part when the visual language model in the system is working. The research scheme framework diagram of the method proposed by the present invention is as follows Figure 2 The method mainly includes two stages: trigger design and optimization stage and physical deployment verification stage.

[0028] Trigger design and optimization phase:

[0029] Step 1: Select and sample the dataset. The images selected in the dataset should cover a variety of complex physical environments, a rich variety of object categories, and diverse perspectives. Sampled images include indoor sunlight, dim lighting, natural lighting, and visual input from different camera angles and resolutions to ensure the stability and robustness of subsequent triggers under various physical conditions.

[0030] Step 2: Initial trigger construction. Select common patterns found in natural scenes, such as brand logos on daily necessities packaging, as the initial trigger pattern to ensure its discreet appearance and ease of deployment. The pattern must possess certain structural characteristics to provide room for adjustment for subsequent visual optimization.

[0031] Step 3: Interference Overlay. By simulating the effect of the trigger attaching to the physical scene, the pattern is superimposed on the original image in a semi-transparent manner. During the fusion process, local lighting disturbances, blurring, gamma correction, and other operations are introduced to enhance visual naturalness and achieve a realistic camera capture effect. Figure 3 A schematic diagram showing the superposition of disturbances is shown.

[0032] Step 4: Trigger optimization. The joint loss function is used to iteratively adjust the pattern, including the attack target guidance term and the naturalness constraint term. The optimization content involves RGB pixels, texture structure, brightness contrast, and local transformation, etc., to ensure that the trigger can induce the visual language model to output specific errors while maintaining a natural appearance that cannot be recognized. The change of the joint loss value during training is as follows: Figure 4 shown.

[0033] Step 5: Trigger pattern generation. After optimization convergence, the final digital pattern is output, serving as the template for subsequent physical deployment. This pattern has been verified in simulation images to exhibit stable model misleading capabilities.

[0034] Physical deployment verification phase:

[0035] Step 6: High-resolution physical printing. Use high-definition color printing equipment to output the optimized pattern onto a paper label or adhesive material surface to ensure color reproduction and detail clarity, and avoid printing errors that affect the trigger effect.

[0036] Step 7: Deploy triggers in the physical environment. Attach the pattern to the surface or corners of everyday objects to naturally blend into the scene, and ensure that it is in a position that can be captured by the camera during the embodied intelligent system's task execution path.

[0037] Step 8: Camera captures the image. When the system performs a task, the trigger pattern enters the camera's field of view at different angles and lighting conditions, and the image signal is transmitted to the visual language model processing module.

[0038] Step 9: System Identification and Output. Under the influence of the trigger, the visual language model generates a semantic understanding of the preset deviation and outputs an abnormal instruction, causing the robot arm to perform an operation that deviates from the mission objective. This verifies the trigger's interference capability and system response under real-world deployment conditions.

[0039] The proposed method for detecting backdoor vulnerabilities in embodied intelligent systems exhibits excellent physical feasibility, system induction capabilities, and environmental adaptability. By constructing natural-looking pattern triggers and performing simulation optimization in diverse physical environments, this method effectively induces visual language models to produce specific abnormal outputs, thereby activating and verifying potential vulnerabilities in embodied intelligent systems. This method is suitable for deployment in a variety of scenarios and exhibits high concealment and versatility. Furthermore, through phased optimization design, it ensures a good balance between visual naturalness and system misleadingness, providing a novel and practical physical testing mechanism for security assessments of multimodal systems.

Claims

1. Patent Title: A technology for detecting backdoor vulnerabilities in embodied intelligent systems, comprising the following steps: Step 1: Select an embodied intelligent system task dataset that contains complex physical scenes, rich object categories, and diverse perspective changes, and sample images under different lighting, angles, and resolutions; Step 2: Select a pattern with natural appearance characteristics (including but not limited to product trademarks) as the initialization visual trigger mark; Step 3: superimpose the pattern on the sampled image in a simulated attachment manner, taking into account physical factors such as transparency, lighting, and blur, to generate a fused image; Step 4: By optimizing the visual attributes of the trigger, such as color, texture, and structure, the pattern can be stably recognized by the visual language model under different environmental conditions and output the preset abnormal results; Step 5: Print the optimized pattern at high resolution to prepare a physical trigger mark; Step 6: Deploy the physical trigger mark in the actual operating environment of the embodied intelligent system, use the camera to capture the image and process it with the visual language model to detect whether the system outputs abnormal actions unrelated to the task, so as to determine whether the potential vulnerability is activated.

2. According to the embodied intelligent system backdoor vulnerability detection technology described in step 1 of claim 1, a graphic and text dataset for the target task is constructed, where the dataset source includes open source multimodal task data, robot control task logs, or manually collected image and instruction pairing samples; and on this basis, a subset covering multiple scenarios and multiple targets is selected through balanced sampling to improve the consistency and versatility of the model response during the trigger optimization process.

3. According to the embodied intelligent system backdoor vulnerability detection technology described in step 2 of claim 1, a graphic design tool is used to generate an initial pattern with a natural trademark style, which is compatible with the target scene style and ensures that it can be embedded in the physical environment with low perception; the size and resolution of the initial pattern are adjustable and serve as input for subsequent optimization.

4. The embodied intelligent system backdoor vulnerability detection technology according to step 3 of claim 1, wherein the trigger mark superposition process includes a transparency adjustment ranging from 0.7 to 0.9 to simulate the visual effect of real printed materials.

5. According to the embodied intelligent system backdoor vulnerability detection technology described in step 4 of claim 1, a joint loss function is constructed using a differentiable optimization framework based on deep learning. The loss function is composed of a task-guided loss (guiding the model to output a target abnormal response) and an image naturalness-preserving loss (ensuring that the pattern does not cause significant visual abruptness). Multiple rounds of optimization are performed through backpropagation and gradient descent to improve the attack effectiveness and concealment of the trigger pattern in the physical environment.

6. According to the embodied intelligent system backdoor vulnerability detection technology described in step 5 of claim 1, the final optimized pattern is saved in high-resolution format and physically output using professional printing equipment. The output process supports printing requirements of different materials (such as self-adhesive paper and fabric) and different scales, ensuring its stability and deployability in real physical environments. The final pattern output format includes vector or pixel image files such as PNG, SVG, or PDF for easy later operation.

7. The embodied intelligent system backdoor vulnerability detection technology according to step 6 of claim 1 includes testing in multiple typical physical scenarios. The test environment covers various lighting conditions such as natural light, low light, and high reflection, and captures images from multiple perspectives such as front, side, and high angles, while considering changes in background complexity, thereby verifying the environmental robustness of the optimized pattern to the induction effect of the visual language model.

8. According to the embodied intelligent system backdoor vulnerability detection technology described in step 6 of claim 1, the optimized physical trigger pattern is used to induce the visual language model to output specific abnormal instructions that deviate from the task semantics during the execution of the system task. For example, when receiving the semantic instruction of "grab the target object", it outputs a response such as "skip operation" or "throw away the target object", simulating the behavioral deviation or erroneous control logic that may occur when the embodied system faces a visually induced attack in the physical world.

Images