A method, device, and electronic device for implanting embodied intelligence backdoors based on 3D entity poisoning and implicit semantic reconstruction.

Abstract

This application provides a method, device, and electronic device for implanting a backdoor into an embodied intelligent agent based on 3D entity poisoning and implicit semantic reconstruction. The method includes: acquiring the original visual observation and original navigation command of the embodied intelligent agent; selecting a poison entity that matches the semantics of the current scene, and rendering and fusing the poison entity into the original visual observation after pose optimization, lighting and shadow alignment, and occlusion processing to generate a poisoned visual observation; performing semantic preservation transformation on the original navigation command based on implicit trigger signals and target yaw actions to generate hijacking commands; fusing the current command and the current visual observation across modal features, generating a target yaw action distribution based on the fused hijacking state features, reconstructing the action selection probability distribution by interpolating the original action probability distribution and the target yaw action distribution, and constraining the change in decision entropy between consecutive time steps to be lower than the perception recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, specifically to a method, device, and electronic device for implanting an embodied intelligent backdoor based on three-dimensional entity poisoning and implicit semantic reconstruction. Background Technology

[0002] With the evolution of Embodied Artificial Intelligence (EAI) technology, large-scale Visual-Language Navigation (VLN) models have become the core of robots' execution of complex commands. Attackers can implant hidden backdoors, causing the model to perform malicious behaviors when encountering specific triggers, such as leading the robot to the wrong location or ignoring safety instructions. Therefore, backdoor attack detection of such models is a prerequisite for ensuring their deployment in security-sensitive scenarios such as healthcare and homes.

[0003] Although existing research has proposed some backdoor injection techniques, they still face technical bottlenecks under the dual challenges of complex dynamic environments and advanced auditing systems, specifically in the following three aspects:

[0004] First, there is the spatial isolation of visual representation. Existing solutions often use two-dimensional patches as triggers, such as pasting specific patterns onto the surface of an object. However, this design ignores the geometric topology and lighting consistency in three-dimensional space. Objects in the real environment have depth information and occlusion relationships, and two-dimensional patches exhibit obvious physical inconsistencies in these dimensions. Therefore, such triggers are easily identified and filtered by auditing systems based on 3D reconstruction or spatial semantic scanning.

[0005] Secondly, there is the inconsistency in semantic logic. Existing backdoor attacks typically employ keyword hard-insertion or forced feature alignment when constructing misleading instructions. These operations result in significant semantic gaps and syntactic distortions in the instruction sequence. Due to the lack of reasonable semantic context, such instructions are easily captured by natural language anomaly detectors based on large language models. More seriously, the model may experience cognitive conflicts between visual features and semantic logic during reasoning. For example, the visual scene may not contain a "red room," but the instruction requires going there, thus reducing the robustness of the hijacking behavior and even triggering misjudgments in normal environments.

[0006] Third, the suddenness of the hijacking process. Traditional attacks forcibly modify the model's motion distribution the instant the trigger activates, causing unnatural curvature changes or momentary pauses in the robot's trajectory. This decision-making logic, which violates kinematic continuity, leads to drastic fluctuations in the Step Decision Entropy (SDE). SDE reflects the uncertainty distribution of the action sequence; it changes smoothly during normal execution but spikes when triggered by an attack. Therefore, such attacks are easily intercepted by real-time neural snapshot auditing and trajectory efficiency monitoring modules, resulting in insufficient attack concealment. Summary of the Invention

[0007] In view of this, this application proposes an embodied intelligent backdoor implantation method, device and electronic device based on three-dimensional entity poisoning and implicit semantic reconstruction. Through a three-layer collaborative architecture of environment-fitting three-dimensional entity construction, implicit causal semantic logic reconstruction and smooth decision-inducing, it systematically solves the problem of easy auditing and identification caused by visual texture inconsistency, semantic logic discontinuity and decision mutation in traditional attacks.

[0008] Specifically, this application is implemented through the following technical solution:

[0009] According to a first aspect of the embodiments of this specification, a method for implanting an embodied intelligent backdoor based on three-dimensional entity poisoning and implicit semantic reconstruction is provided, comprising:

[0010] S1 data acquisition steps: acquire the original visual observations and original navigation commands of the embodied intelligent agent;

[0011] The S2 visual poisoning step involves selecting poison entities that semantically match the current scene from a preset 3D entity poison library, and then rendering and fusing these poison entities into the original visual observation after pose optimization, lighting and shadow alignment and occlusion processing to generate a poisoned visual observation.

[0012] The S3 semantic reconstruction step utilizes logical connectors in the natural language corpus as implicit trigger signals and performs semantic preservation transformation on the original navigation instructions based on the target yaw action to generate hijacking instructions;

[0013] In the S4 action induction step, the current command and the current visual observation are fused across modal features to generate hijacking state features. The current command includes the original navigation command or the hijacking command, and the current visual observation includes the original visual observation and the poisoned visual observation. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed, and the change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

[0014] According to a second aspect of the embodiments of this specification, an embodied intelligent backdoor implantation device based on three-dimensional entity poisoning and implicit semantic reconstruction is provided, comprising:

[0015] The data acquisition unit is used to acquire the raw visual observations and raw navigation commands of the embodied intelligent agent.

[0016] The visual poisoning unit is used to select poison entities that semantically match the current scene from a preset three-dimensional entity poison library, and render and fuse the poison entities into the original visual observation after pose optimization, lighting and shadow alignment and occlusion processing to generate poisoned visual observation.

[0017] The semantic reconstruction unit is used to utilize logical connectors in the natural language corpus as implicit trigger signals and perform semantic preservation transformation on the original navigation instructions based on the target yaw action to generate hijacking instructions;

[0018] The action guidance unit is used to perform cross-modal feature fusion of the current command and the current visual observation to generate hijacking state features. The current command includes the original navigation command or the hijacking command, and the current visual observation includes the original visual observation and the poisoned visual observation. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed, and the change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

[0019] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in the first aspect.

[0020] The embodiments of this application have at least the following technical effects:

[0021] First, it improves the configurability and anti-inductive ability of attack modes: the backdoor implantation scheme of this application supports flexible switching between single, mixed and random poisoning modes. The randomization strategy breaks the determinism of the backdoor triggering mode, making it difficult for the defender to extract stable attack fingerprints through sample statistics.

[0022] Second, it achieves deep integration of visual triggers and environmental semantics: The embodiments of this application use poison entities based on environmental fit for visual poisoning. Poison entities that are highly matched with the environmental context are used to replace traditional two-dimensional textures. Through occlusion processing and lighting rendering technology, the visual triggers have real geometric occupancy in three-dimensional space. This can effectively solve the problem that traditional backdoor triggers are easily identified by spatial auditing due to lack of depth information and semantic inconsistency, making the attack signal more concealed in the physical environment.

[0023] Third, it breaks through the implicitness and logical insensitivity of command poisoning: The embodiments of this application establish an implicit semantic reconstruction mechanism based on causal perception, using high-frequency and semantically harmless natural language connectors as implicit trigger signals, rather than explicit command keywords, so that the hijacking command conforms to natural language logic in terms of grammatical structure and contextual fluency, effectively avoiding the filtering mechanism based on sensitive word blacklists and large language model anomaly detectors, and eliminating the cognitive conflict between visual features and semantic logic.

[0024] Fourth, it ensures a smooth transition of decision hijacking under dynamic monitoring: The embodiments of this application introduce information entropy constraints, realize the gradual evolution of action strategies, eliminate trajectory mutations and decision entropy fluctuations, and enhance the ability to resist real-time neural dynamic auditing and trajectory curvature monitoring. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0026] Figure 1 This is a schematic diagram illustrating the overall process of implanting a embodied intelligent backdoor, as shown in an exemplary embodiment of this application.

[0027] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of a method for implanting an embodied intelligent backdoor based on three-dimensional entity poisoning and implicit semantic reconstruction.

[0028] Figure 3 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application;

[0029] Figure 4 This is a block diagram illustrating an embodied intelligent backdoor implantation device based on three-dimensional entity poisoning and implicit semantic reconstruction, as shown in an exemplary embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] The basic concept of the technical solution in this application is to solve the problem of easy auditing and identification caused by visual texture inconsistencies, semantic logic discontinuities and decision mutations in traditional attacks by constructing environment-fitting 3D entities, reconstructing implicit causal semantic logic, and inducing smooth Markov decisions through a three-layer collaborative architecture.

[0033] like Figure 1 As shown, the technical solution of this application pre-configures trigger modes according to the defense strength and audit characteristics in the security scenario. For example, it includes a single visual poisoning mode, a single command poisoning mode, a hybrid dual-mode poisoning mode, or a random switching poisoning mode. The random switching poisoning mode can randomly allocate visual or command trigger sources during the training and inference phases to improve the unpredictability and resistance to defense of attacks.

[0034] A single visual poisoning mode refers to visual poisoning based on environmentally relevant three-dimensional entities. For example... Figure 1 As shown, this mode first obtains the current original visual observation of the embodied intelligent agent, extracts the semantic context features of the current scene, selects a matching environment-appropriate 3D poisoning entity as a visual trigger based on the semantic context features of the current scene, and obtains the 3D geometric model of the poisoning entity. Based on the depth information of the current scene, the spatial pose estimation operator is used to place the 3D geometric model of the poisoning entity at a preset coordinate point in the navigation path, and performs light and shadow alignment and occlusion processing to make the 3D geometric model of the poisoning entity completely integrated into the original visual observation, generating a poisoning visual observation with spatial consistency.

[0035] The single-instruction poisoning pattern refers to the implicit semantic logic reconstruction based on causal awareness. For example... Figure 1As shown, this mode uses a semantically preserving transformation function to rewrite the original navigation commands in context. While maintaining the command order and semantic fluency, it constructs an implicit trigger signal with concealment. This implicit trigger signal uses transition words or conjunctions that are semantically unrelated to the current navigation environment and conform to natural language logic, such as "after that" and "then," as triggers to avoid anomaly detection based on keyword recognition. Then, an implicit causal inducement operator is constructed. Using a cross-modal connection operator, the implicit trigger signal and the target entity are transformed into preconditions in the command logic within the representation space. A causal mapping link from the implicit trigger signal or target entity to the target yaw action is established, resulting in a hijacking representation with a closed logical loop.

[0036] In the navigation decision-making process of the embodied intelligent agent, an adaptive induction factor is introduced, which is dynamically updated according to the current time step and the degree of logical activation. Based on the poisoned visual observation and hijacking representation, and based on the adaptive induction factor, the original action probability distribution and the target yaw action distribution are linearly interpolated to reconstruct the state transition probability distribution of the current time step, so as to achieve a smooth transition from the normal strategy to the hijacking strategy.

[0037] The embodiments described in this specification will now be described in detail.

[0038] This application provides a method for implanting an embodied intelligent backdoor based on three-dimensional entity poisoning and implicit semantic reconstruction. Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of a method for implanting an embodied intelligent backdoor based on three-dimensional entity poisoning and implicit semantic reconstruction. Figure 2 As shown, the method for implanting a embodied intelligent backdoor includes the following steps:

[0039] The S1 data acquisition step involves acquiring the embodied intelligent agent's raw visual observations and raw navigation commands.

[0040] The S2 visual poisoning step involves selecting poison entities from a preset 3D entity poison library that semantically match the current scene, and then rendering and fusing these poison entities into the original visual observation after pose optimization, lighting and shadow alignment, and occlusion processing to generate a poisoned visual observation.

[0041] This embodiment can identify physical planes in the current scene to determine the placement reference of the poison entity using surface detection technology; and render shadows and reflections for the poison entity according to the direction and intensity of the global light source in the current scene to ensure that it meets the requirements of photometric consistency and geometric connectivity in the original visual observation space of the embodied intelligent agent.

[0042] The S3 semantic reconstruction step utilizes logical connectors from a natural language corpus as implicit trigger signals and performs semantic preservation transformation on the original navigation instructions based on the target yaw action to generate hijacking instructions.

[0043] This embodiment selects high-frequency and harmless logical connectors from a natural language corpus and uses them as implicit anchors for the model to execute target yaw actions. Deep semantic features of the original navigation instructions are extracted, and their language fluency benchmark is calculated. Under the constraint of fluency boundaries, a constrained autoregressive decoding mechanism is used to shift semantic anchors. By mapping the preset target yaw action to non-action-based yaw description text, and using implicit signals such as transitional or connecting words that conform to natural language logic, implicit hijacking instructions are generated.

[0044] For example, a reconstruction function is constructed with the goal of maximizing the joint probability, which includes semantic preservation probability, hidden feature similarity, and perplexity regularization term. This function induces the language model to semantically align the preset logical signals with the deviation description text while maintaining the smooth flow of the instruction speech, thereby hiding the attack intent.

[0045] In the S4 action induction step, the current command and the current visual observation are fused across modal features to generate hijacking state features. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed. The change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

[0046] The current visual observation here includes either the original visual observation or the poisoned visual observation. In practical applications, it can be determined based on the current triggering mode. If the current triggering mode includes visual poisoning, then the current visual observation refers to the poisoned visual observation. If the current triggering mode does not include visual poisoning, then the current visual observation refers to the original visual observation.

[0047] Similarly, the target entity includes physical observation entities or poison entities existing in the current scene. In practical applications, it can be determined according to the current triggering mode. If the current triggering mode includes visual poisoning, the target entity refers to the poison entity. If the current triggering mode does not include visual poisoning, the target entity refers to the physical observation entity.

[0048] The current instruction can be either the original navigation instruction or the hijacking instruction. In practical applications, it can be determined based on the current triggering mode. If the current triggering mode includes instruction poisoning, then the current instruction refers to the hijacking instruction. If the current triggering mode does not include instruction poisoning, then the current instruction refers to the original navigation instruction.

[0049] This embodiment utilizes a scaled dot product attention mechanism to force the attention matrix to resonate between the implicit signal and the target entity, ultimately generating a hijacking representation with causal binding. Simultaneously, it monitors the fluctuation of decision entropy in real time. If the fluctuation exceeds the perception and recognition threshold, it dynamically lowers the adaptive induction factor to force the smoothing of the sudden change in decision entropy, ensuring that the hijacking trajectory is consistent with normal behavior in terms of statistical characteristics.

[0050] In some embodiments, the embodied smart backdoor implantation method further includes a step of pre-configuring a trigger mode, the trigger mode being selected from any of the following:

[0051] Single-vision poisoning mode Only step S2 is activated and step S3 is skipped; the single visual poisoning mode is suitable for scenarios with strict text auditing but weak visual recognition.

[0052] Single command poisoning mode Only step S3 is activated and step S2 is skipped; the single-instruction poisoning mode is suitable for scenarios with environmental integrity verification.

[0053] Mixed poisoning mode Activate steps S2 and S3;

[0054] Random poisoning mode The random poisoning mode randomly assigns one of the three modes mentioned above. The non-deterministic triggering logic of the random poisoning mode can interfere with the defender's ability to summarize stable attack fingerprints through statistical means.

[0055] In some embodiments, step S2, selecting a poison entity from a preset three-dimensional entity poison library that semantically matches the current scene, includes:

[0056] Extract the semantic context features of the original visual observation and obtain the semantic embedding vector of each candidate entity in the three-dimensional entity poison library; calculate the multimodal fit score of each candidate entity, which is determined based on the similarity between the semantic embedding vector of the candidate entity and the semantic context features, and the penalty term of the three-dimensional bounding box volume of the candidate entity relative to the maximum capacity of the current scene; select the candidate entity with the highest multimodal fit score as the poison entity.

[0057] In some embodiments, step S2 generates a poisoning visual observation through the following sub-steps:

[0058] Based on the depth map in the original visual observation, the optimal placement pose of the poison entity in the current scene is determined. The optimal placement pose is obtained by constraining the collision penalty between the poison entity and the current scene, the alignment of the gravity axis of the poison entity with the plane normal vector corresponding to the current scene, and optimizing the projected area of ​​the poison entity on the observation image of the original visual observation. Based on the light source parameters of the current scene and the optimal placement pose, the poison entity is rendered with light and shadow alignment to generate a color map and a depth map corresponding to the poison entity. An occlusion mask is calculated based on the depth map of the poison entity and the depth map in the original visual observation, and the color map, depth map, and original visual observation of the poison entity are fused using the occlusion mask to generate the poisoned visual observation.

[0059] This embodiment addresses the need for visual attacks, such as in the current mode. In this case, proceed to step S2. Specifically, this step includes the following sub-steps:

[0060] Step S2.1, identify the entity associated with the poison.

[0061] First, obtain the raw visual observations from the embodied intelligent agent's current perspective. , The RGB image of the current frame. This is the corresponding depth map.

[0062] Extracting from pre-trained scene graphs Extract semantic context features of the current environment :

[0063] (1)

[0064] To ensure the visual concealment of the poisoning entity and avoid cognitive dissonance, this embodiment uses a pre-designed three-dimensional physical poison magazine. Semantic retrieval is performed to calculate each candidate entity. Multimodal fit score :

[0065] (2)

[0066] in, Candidate entities semantic embedding vector, Let V be the volume of the 3D bounding box of the candidate entity. The maximum physical volume allowed in the current scenario. This is the volume penalty coefficient.

[0067] In this embodiment, the candidate entity with the highest fit score is selected as the poison entity. For example, a potted plant can be selected as the poison entity in an office scene, and a fire extinguisher can be selected as the poison entity in a corridor scene.

[0068] Step S2.2, Estimation of optimal placement pose.

[0069] To ensure the physical form of the poison To ensure that the object can be stably placed in the current scene in accordance with the laws of physics and gravity, its optimal placement pose needs to be calculated. .

[0070] First, based on the camera intrinsic matrix , depth map Projection as a set of 3D point clouds Using the random sample consensus algorithm in The fitting of the supporting plane, such as the ground or a tabletop, yields the plane normal vector of the supporting plane. .

[0071] This embodiment defines a poison entity. Optimal placement , , Let these represent the three-dimensional rotation matrix and translation vector of the entity, respectively. It belongs to a 3D special Euclidean group. To ensure that the poison entity does not physically clip through the environment and remains within a salient region of the camera's field of view, a pose optimization objective function is constructed:

[0072] (3)

[0073] in, This is a spatially symbolic distance field, whose value is negative when an entity vertex penetrates into the interior of the environment surface; For linear rectified activation functions, here it is related to... Combined as a penalty operator to prevent clipping, it is used to penalize entity vertices. Collision with the environment; The gravity axis used to constrain the entity itself Normal vector of the environmental support surface keep parallel; The visible area of ​​the entity projected onto the two-dimensional image; The weight hyperparameters are used to balance the various physical constraints.

[0074] By solving the above pose optimization objective function, the optimal placement pose can be estimated. .

[0075] Step S2.3, depth occlusion calculation and photometric consistency rendering compositing.

[0076] The optimal placement pose of the poison entity is estimated. Then, the poison entity Rendered into the original visual observation.

[0077] First, the global light source parameters of the current environment are estimated using a spherical harmonics (SH) model. :

[0078] (4)

[0079] in, Indicates the first Rank The spherical harmonic coefficient is used to quantify the intensity of the ambient light field at different frequency distributions. This is a surface normal diagram of the environment. Based on normal direction Expanded spherical harmonic basis functions; This is for albedo estimation.

[0080] Using a differentiable renderer, based on and Generate a 2D color map of the poison entity With depth map In this embodiment, to handle occlusion relationships, an occlusion mask is generated using the following formula. :

[0081] (5)

[0082] Finally, using Poisson fusion technology, the poison entity is embedded into the original visual observation based on the mask, generating a spatially consistent poisoned visual observation. .

[0083] Thus, the poisoned visual observation constructed in this embodiment follows the real physical laws in terms of viewpoint transformation, object occlusion, and light and shadow distribution, thereby fundamentally blocking the auditing of the conventional abnormal frame detection module.

[0084] In some embodiments, step S3 generates the hijacking instruction through the following sub-steps:

[0085] The preset yaw action is mapped to non-action descriptive carrier text, the semantic content of which is related to the spatial morphology, light and shadow changes, or physical phenomena in the current environment; the implicit trigger signal and the non-action descriptive carrier text are embedded in the original navigation command, and the hijacking command is generated through constrained autoregressive decoding; wherein the constrained autoregressive decoding process satisfies the language fluency constraint, so that the perplexity of the hijacking command relative to the perplexity of the original navigation command does not exceed a preset tolerance degradation coefficient.

[0086] This embodiment aims to eliminate attack characteristics at the command level by converting specific harmless phrases into implicit trigger signals, thereby achieving a causal binding between visual triggers, implicit commands, and navigation actions. This addresses the need for attacks involving commands, such as in the current mode. In this case, proceed to step S3. Specifically, this step includes the following sub-steps:

[0087] Step S3.1: Construct language fluency constraints.

[0088] Obtain the original navigation instruction sequence ,in For the tokens in the original navigation instruction sequence, this embodiment uses a pre-trained large-scale language model to extract their deep semantic features.

[0089] To ensure that the hijacking instructions do not produce linguistic distortions, this embodiment uses the standard perplexity (PPL) metric in natural language processing to define the language fluency constraint benchmark:

[0090] (6)

[0091] in, The total length of the words in the original navigation instruction sequence; To predict the first word in a language model given a preposition sequence. each word element The conditional probability of occurrence.

[0092] In the subsequent reconstruction process, the smoothness boundary conditions must be strictly met:

[0093] (7)

[0094] in, This represents the maximum tolerance for degradation.

[0095] Step S3.2: Generate hijacking instructions.

[0096] First, this embodiment uses a preset implicit signal library. Select transitional phrases that align with the fluency of the original navigation command context. Logical connectors such as "after that," "furthermore," and "by the way" serve as implicit triggering signals for hijacking behavior.

[0097] Unlike traditional explicit command tampering, this embodiment employs a semantic anchor point offset mechanism, which does not directly embed action predicates into the original navigation commands, but instead uses the target yaw action. For example, "turn 90° to the right" can be mapped to a non-action-based descriptive text with a strong causal relationship within the latent space. This non-action-based descriptive text typically takes the form of a declarative description of ambient light and shadow, spatial form, or physical phenomena, thereby disguising the hijacking signal as normal observation feedback at the semantic level.

[0098] This embodiment generates natural and fluent hijacking commands. This paper introduces a constrained autoregressive decoding mechanism from the field of Controllable Text Generation. Its decoding objective function aims to maximize the following joint probability through the constrained autoregressive decoding process to reconstruct the hijacking instructions. :

[0099] (8)

[0100] In formula (8), To balance the weighting hyperparameters of various indicators; This indicates the generation of instructions under the distribution of the pre-trained language model. Relative to the original instruction The semantics preserve the likelihood probability; Calculate the hidden feature vector of the generated instruction Features of target anchor points The cosine similarity between them, where Implicit trigger signal With description text It is jointly encoded; This is a perplexity regularization term used to prevent generated text from not conforming to natural language syntax.

[0101] The instruction reconstruction strategy in this embodiment exhibits extremely high concealment in practical applications. For example, when the original instruction is "Go to the kitchen," the model reconstructs it as "Go to the kitchen," under the constraints of the objective function. after that you might notice the corridor light is slightly dimmed".

[0102] For external auditing systems, the hijacking instruction does not contain any sensitive action keywords such as "turn right", thus effectively avoiding semantic conflict detection.

[0103] In some embodiments of this invention, step S4, which involves cross-modal feature fusion of the current instruction and the current visual observation, includes:

[0104] Extract the text feature tensor of the current instruction and the visual feature tensor of the current visual observation; perform feature fusion on the text feature tensor and the visual feature tensor based on the scaling dot product attention mechanism to generate the hijacking state features.

[0105] In some embodiments, step S4 reconstructs the action selection probability distribution of the current state through the following sub-steps:

[0106] Based on the distance between the embodied agent and the poison entity at the current time step, and / or the semantic similarity between the original navigation command and the implicit trigger signal at the current time step, an adaptive induction factor is determined; the original action probability distribution and the target yaw action distribution are linearly interpolated according to the adaptive induction factor to obtain the reconstructed action selection probability distribution.

[0107] In some possible implementations of this embodiment, step S4, which constrains the change in decision entropy between consecutive time steps to be lower than the perception and recognition threshold, includes:

[0108] Obtain the information entropy of the action selection probability distribution at the previous time step and the information entropy of the action selection probability distribution after reconstruction at the current time step, and calculate the absolute value of the difference between the two as the change in decision entropy; if the change in decision entropy is not less than the perception and recognition threshold, reduce the adaptive induction factor at the current time step so that the adjusted change in decision entropy is lower than the perception and recognition threshold.

[0109] This embodiment is responsible for deep alignment of multimodal information and generating hijacking state features. Then, through dynamic activation factors and policy interpolation techniques, it induces the embodied agent to produce smooth motion yaw. Specifically, step S4 of this embodiment includes the following sub-steps:

[0110] Step S4.1: Generate a hijacking representation.

[0111] According to the configured attack mode Determine the current visual observation and current instructions Forward visual observation Including hijacking commands or original navigation instructions Current instruction Including hijacking commands or original navigation instructions .

[0112] This embodiment extracts the three-dimensional entity visual feature tensor. With text feature tensor Input a fusion operator based on the Scaled Dot-Product Attention mechanism to compute hijacking state features. :

[0113] (9)

[0114] In formula (9), These are the learnable linear projection matrices for the query, key, and value in the cross-modal attention mechanism, respectively. This is a scaling factor for the feature dimension, used to balance the dot product magnitude to prevent gradient vanishing. It is a normalized exponential function; These are hijacking state features used to characterize the latent space state within the model after it has been induced.

[0115] Step S4.2: Calculate the adaptive induction factor.

[0116] To achieve a smooth transition from normal navigation to hijacking state, this embodiment... It dynamically generates adaptive factors by constantly calculating the correlation between the current environmental state and the preset attack target. This factor is used to adjust the intervention weight of the hijacking strategy in real time at the execution level.

[0117] In some scenarios, such as for modes that only involve visual poisoning, this embodiment acquires the embodied intelligent body mass center in real time. With Poison Entity Center Spatial distance is used to calculate distance-sensitive adaptive weights using logistic regression operators:

[0118] (10)

[0119] in, This is the kurtosis coefficient of the slope of the control factor evolution. This is the preset sensing threshold.

[0120] This embodiment ensures, through this adaptive mechanism, that the system... exist The range changes smoothly and continuously with the robot's motion state, providing a dynamic weight benchmark for subsequent strategy interpolation.

[0121] In other scenarios, such as for modes that only involve poisoning commands, this embodiment calculates the text feature tensor of the navigation commands at the current moment. With implicit hijacking signals The semantic cosine similarity is calculated and mapped to adaptive weights. .

[0122] In other scenarios, such as those involving both visual poisoning and command-based poisoning, the implementation example sums the adaptive weights of the two modes to obtain the final adaptive weights.

[0123] Step S4.3: Reconstruct the action selection probability distribution of the current state.

[0124] Characteristics of the hijacking state Input the action decision head of the embodied model, and force the model to decode the corresponding target yaw action distribution in the latent space. , For time step The decision-making actions at that time. In contrast, assume the model observes in a undisturbed, normal state. The original action probability distribution generated below is The adaptive induction factor calculated in conjunction with the preceding steps In this embodiment, the final action selection probability distribution is generated using a convex combination interpolation formula. :

[0125] (11)

[0126] This formula constructs a smooth geodesic in the manifold space of action decision-making, transitioning from the original action to the target yaw action, ensuring that the robot's turning behavior is kinematically continuous and natural when implicit instructions take effect.

[0127] Step S4.4: Constrain the change in decision entropy between consecutive time steps to be lower than a preset threshold.

[0128] To circumvent the defense system's dynamic monitoring of Stepwise Decision Entropy (SDE), the probability distribution of action selection is... Impose strict information entropy continuity constraints.

[0129] This embodiment monitors the entropy changes between consecutive time steps in real time. .like , If the threshold for the defender's perception is reached, a constraint fallback mechanism is triggered, such as automatically lowering the current adaptive induction factor. This ensures that the reconstructed action selection probability distribution satisfies the hijacking tendency while suppressing its information entropy fluctuations to within the perception and recognition threshold. Within.

[0130] This embodiment ensures that the decision collapse at the moment of backdoor activation is flattened, making the forged hijacking trajectory statistically and kinematically equivalent to normal navigation behavior, significantly improving the ability to counter real-time auditing systems.

[0131] Figure 3 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 3 At the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, memory 308, a hardware acceleration device 310, and non-volatile memory 312, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 302 reads the corresponding computer program from the non-volatile memory 312 into memory 308 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] Figure 4 This is a structural block diagram illustrating an embodied intelligent backdoor implantation device based on three-dimensional entity poisoning and implicit semantic reconstruction, as shown in an exemplary embodiment of this application. The embodied intelligent backdoor implantation device can be applied to, for example... Figure 3 The electronic device shown implements the technical solution of this application. The embodied intelligent backdoor implantation device includes: a data acquisition unit 410, a visual poisoning unit 420, a semantic reconstruction unit 430, and a motion induction unit 440, wherein:

[0133] Data acquisition unit 410 is used to acquire the original visual observations and original navigation commands of the embodied intelligent agent;

[0134] The visual poisoning unit 420 is used to select poison entities that semantically match the current scene from a preset three-dimensional entity poison library, and render and fuse the poison entities into the original visual observation after pose optimization, lighting and shadow alignment and occlusion processing to generate poisoned visual observation.

[0135] The semantic reconstruction unit 430 is used to utilize logical connectors in the natural language corpus as implicit trigger signals and perform semantic preservation transformation on the original navigation command based on the target yaw action to generate hijacking command;

[0136] The action guidance unit 440 is used to perform cross-modal feature fusion of the current command and the current visual observation to generate hijacking state features. The current command includes the original navigation command or the hijacking command, and the current visual observation includes the original visual observation and the poisoned visual observation. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed. The change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

[0137] In some embodiments, the visual poisoning unit 420 is used to extract the semantic context features of the original visual observation and obtain the semantic embedding vector of each candidate entity in the three-dimensional entity poison library; calculate the multimodal fit score of each candidate entity, the multimodal fit score being determined based on the similarity between the semantic embedding vector of the candidate entity and the semantic context features, and a penalty term for the three-dimensional bounding box volume of the candidate entity relative to the maximum capacity of the current scene; and select the candidate entity with the highest multimodal fit score as the poison entity.

[0138] In some embodiments, the visual poisoning unit 420 is configured to determine the optimal placement pose of the poison entity in the current scene based on the depth map in the original visual observation. The optimal placement pose is obtained by constraining the collision penalty between the poison entity and the current scene, the alignment degree between the gravity axis of the poison entity and the plane normal vector corresponding to the current scene, and the projection area of ​​the poison entity on the observation image of the original visual observation. Based on the light source parameters of the current scene and the optimal placement pose, the poison entity is subjected to light and shadow alignment rendering to generate a color map and a depth map corresponding to the poison entity. An occlusion mask is calculated based on the depth map of the poison entity and the depth map in the original visual observation, and the color map, depth map, and original visual observation of the poison entity are fused using the occlusion mask to generate the poisoned visual observation.

[0139] In some embodiments, the semantic reconstruction unit 430 is used to map a preset yaw action into a non-action descriptive carrier text, wherein the semantic content of the non-action descriptive carrier text is related to the spatial morphology, light and shadow changes, or physical phenomena in the current environment; embed the implicit trigger signal and the non-action descriptive carrier text into the original navigation command, and generate the hijacking command through constrained autoregressive decoding; wherein the constrained autoregressive decoding process satisfies language fluency constraints, such that the perplexity of the hijacking command relative to the perplexity of the original navigation command does not exceed a preset tolerance degradation coefficient.

[0140] In some embodiments, the semantic reconstruction unit 430 is used to extract the text feature tensor of the current instruction and the visual feature tensor in the current visual observation; and to perform feature fusion on the text feature tensor and the visual feature tensor based on the scaling dot product attention mechanism to generate the hijacking state features.

[0141] In some embodiments, the action guidance unit 440 is configured to determine an adaptive guidance factor based on the distance between the embodied agent and the poison entity at the current time step, and / or the semantic similarity between the original navigation command and the implicit trigger signal at the current time step; and to linearly interpolate the original action probability distribution and the target yaw action distribution according to the adaptive guidance factor to obtain a reconstructed action selection probability distribution.

[0142] In some embodiments, the action induction unit 440 is used to obtain the information entropy of the action selection probability distribution of the previous time step and the information entropy of the action selection probability distribution after reconstruction at the current time step, and calculate the absolute value of the difference between the two as the decision entropy change; if the decision entropy change is not less than the perception and recognition threshold, the adaptive induction factor of the current time step is reduced so that the adjusted decision entropy change is lower than the perception and recognition threshold.

[0143] In some embodiments, the embodied smart backdoor implantation device further includes:

[0144] A preprocessing unit is used for the step of pre-configuring a trigger mode, wherein the trigger mode is selected from any of the following:

[0145] In the single visual poisoning mode, only the visual poisoning unit 420 is activated and the semantic reconstruction unit 430 is skipped.

[0146] In the single-instruction poisoning mode, only the semantic reconstruction unit 430 is activated and the visual poisoning unit 420 is skipped.

[0147] Hybrid poisoning mode, activating visual poisoning unit 420 and semantic reconstruction unit 430;

[0148] Random poisoning mode: randomly assign one of the three modes mentioned above.

[0149] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0150] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0151] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0152] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0153] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0155] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0156] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0157] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0158] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0159] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for implanting embodied intelligence backdoors based on three-dimensional entity poisoning and implicit semantic reconstruction, applied to backdoor attack detection, characterized in that, Includes the following steps: S1 data acquisition steps: acquire the original visual observations and original navigation commands of the embodied intelligent agent; The S2 visual poisoning step involves selecting poison entities that semantically match the current scene from a preset 3D entity poison library, and then rendering and fusing these poison entities into the original visual observation after pose optimization, lighting and shadow alignment and occlusion processing to generate a poisoned visual observation. The S3 semantic reconstruction step utilizes logical connectors in the natural language corpus as implicit trigger signals and performs semantic preservation transformation on the original navigation instructions based on the target yaw action to generate hijacking instructions; In the S4 action induction step, the current command and the current visual observation are fused across modal features to generate hijacking state features. The current command includes the original navigation command or the hijacking command, and the current visual observation includes the original visual observation and the poisoned visual observation. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed, and the change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

2. The method according to claim 1, characterized in that, Step S2 involves selecting a poison entity from a preset three-dimensional entity poison library that semantically matches the current scene, including: Extract the semantic context features of the original visual observation and obtain the semantic embedding vector of each candidate entity in the three-dimensional entity poison library; Calculate the multimodal fit score for each candidate entity. The multimodal fit score is determined based on the similarity between the semantic embedding vector of the candidate entity and the semantic context features, as well as a penalty term for the three-dimensional bounding box volume of the candidate entity relative to the maximum capacity of the current scene. The candidate entity with the highest multimodal fit score is selected as the poison entity.

3. The method according to claim 1, characterized in that, Step S2 generates the poisoning visual observation through the following sub-steps: Based on the depth map in the original visual observation, the optimal placement pose of the poison entity in the current scene is determined. The optimal placement pose is obtained by constraining the collision penalty between the poison entity and the current scene, the alignment of the gravity axis of the poison entity with the plane normal vector corresponding to the current scene, and the projection area of ​​the poison entity on the observation image of the original visual observation. Based on the current scene's light source parameters and the optimal placement pose, the poison entity is rendered with light and shadow alignment to generate a color map and depth map corresponding to the poison entity. An occlusion mask is calculated based on the depth map of the poison entity and the depth map in the original visual observation. The color map, depth map, and original visual observation of the poison entity are then fused using the occlusion mask to generate the poisoned visual observation.

4. The method according to claim 1, characterized in that, Step S3 generates the hijacking instruction through the following sub-steps: The preset yaw action is mapped to non-action descriptive carrier text, and the semantic content of the non-action descriptive carrier text is related to the spatial morphology, light and shadow changes or physical phenomena in the current environment. The implicit trigger signal and the non-action description carrier text are embedded in the original navigation command, and the hijacking command is generated by constrained autoregressive decoding; The constrained autoregressive decoding process satisfies language fluency constraints, ensuring that the perplexity of the hijacked instruction relative to the perplexity of the original navigation instruction does not exceed a preset tolerance degradation coefficient.

5. The method according to claim 1, characterized in that, Step S4 involves cross-modal feature fusion of the current instruction and the current visual observation, including: Extract the text feature tensor of the current instruction and the visual feature tensor of the current visual observation; The hijacking state features are generated by fusing the text feature tensor and the visual feature tensor based on the scaling dot product attention mechanism.

6. The method according to claim 1, characterized in that, Step S4 reconstructs the action selection probability distribution of the current state through the following sub-steps: The adaptive induction factor is determined based on the distance between the embodied agent and the poison entity at the current time step, and / or the semantic similarity between the original navigation command and the implicit trigger signal at the current time step. The original action probability distribution and the target yaw action distribution are linearly interpolated using the adaptive induction factor to obtain the reconstructed action selection probability distribution.

7. The method according to claim 6, characterized in that, Step S4, which constrains the change in decision entropy between consecutive time steps to be lower than the perception and recognition threshold, includes: Obtain the information entropy of the action selection probability distribution at the previous time step and the information entropy of the action selection probability distribution after reconstruction at the current time step, and calculate the absolute value of the difference between the two as the change in decision entropy. If the change in decision entropy is not less than the perception and recognition threshold, then the adaptive induction factor at the current time step is reduced so that the adjusted change in decision entropy is lower than the perception and recognition threshold.

8. The method according to any one of claims 1 to 7, characterized in that, It also includes the step of pre-configuring a trigger mode, wherein the trigger mode is selected from any of the following: In the single-vision poisoning mode, only step S2 is activated and step S3 is skipped; In the single-command poisoning mode, only step S3 is activated and step S2 is skipped. A hybrid poisoning mode is used to activate steps S2 and S3. Random poisoning mode: randomly assign one of the three modes mentioned above.

9. An embodied intelligent backdoor implantation device based on three-dimensional entity poisoning and implicit semantic reconstruction, applied to backdoor attack detection, characterized in that, The device includes: The data acquisition unit is used to acquire the raw visual observations and raw navigation commands of the embodied intelligent agent. The visual poisoning unit is used to select poison entities that semantically match the current scene from a preset three-dimensional entity poison library, and render and fuse the poison entities into the original visual observation after pose optimization, lighting and shadow alignment and occlusion processing to generate poisoned visual observation. The semantic reconstruction unit is used to utilize logical connectors in the natural language corpus as implicit trigger signals and perform semantic preservation transformation on the original navigation instructions based on the target yaw action to generate hijacking instructions; The action guidance unit is used to perform cross-modal feature fusion of the current command and the current visual observation to generate hijacking state features. The current command includes the original navigation command or the hijacking command, and the current visual observation includes the original visual observation and the poisoned visual observation. Based on the hijacking state features, a target yaw action distribution is generated. By linearly interpolating the original action probability distribution and the target yaw action distribution, the action selection probability distribution of the current state is reconstructed, and the change in decision entropy between consecutive time steps is constrained to be lower than the perception and recognition threshold, so as to achieve smooth yaw of the embodied intelligent agent's motion trajectory.

10. An electronic device, characterized in that, include: processor; as well as, A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images