Robot control method and system based on semantic embedding and attention bi-consistency, terminal and storage medium

Abstract

The application relates to the technical field of intelligent control and discloses a robot control method and system based on semantic embedding and attention consistency, a terminal and a storage medium, the method comprising the following steps: acquiring an action sequence of a target robot; analyzing real multi-modal semantic representations, real attention distributions, simulated multi-modal semantic representations and simulated attention distributions according to the action sequence, and constructing a loss function based on the real multi-modal semantic representations, the real attention distributions, the simulated multi-modal semantic representations and the simulated attention distributions, so as to train a student branch in a vision-language-action model; inputting a test disturbed image and a test language instruction into the trained student branch, and outputting a control instruction for controlling the target robot. The application realizes explicit constraints of attention consistency and patch suppression, makes cross-modal attention converge from a patch region to a real target region related to a task, reduces interference of an adversarial patch on attention logic of the model, significantly improves robustness of the model by adopting a light-weight fine-tuning strategy, and improves a success rate of task execution.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a robot control method, system, terminal, and computer-readable storage medium based on semantic embedding and attentional consistency. Background Technology

[0002] Embodied intelligence VLA models (Vision-Language-Action Models) jointly model visual input and natural language commands, and directly output robot action sequences. These end-to-end models are increasingly being used in complex scenarios such as industrial production and warehousing logistics for tasks such as grasping, handling, and manipulation.

[0003] However, due to the black-box nature of VLA models, the output is executed directly. Small deviations in perception and alignment may be amplified along the "vision-language-action" link into action trajectory deviations and task failures. Existing classification and recognition tasks introduce teacher-student distillation and maintain attention consistency, mainly for cross-modal (text to vision) information complementarity. The aim is to improve the accuracy of group behavior recognition in a clean environment, but it cannot solve the problem in the VLA domain when the model is subjected to adversarial attacks.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a robot control method, system, terminal, and computer-readable storage medium based on semantic embedding and attentional consistency. This invention aims to address the problems in the existing teacher-student distillation model, such as its inability to achieve robust defense against attacks on VLA domain models due to its poor fine-tuning multi-focus efficiency and decoding structure, which easily leads to errors in action execution.

[0006] To achieve the above objectives, this invention provides a robot control method based on semantic embedding and attentional consistency, the robot control method based on semantic embedding and attentional consistency comprising the following steps:

[0007] Acquire natural language commands input by the user, and acquire multiple observation images collected by the target robot based on the natural language commands;

[0008] A vision-language-action model is constructed, and all the observed images and natural language commands are input into the vision-language-action model. The model outputs a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution.

[0009] A semantic consistency loss function is constructed using the real multimodal semantic representation and the simulated multimodal semantic representation, and an attention consistency loss function is constructed using the real attention distribution and the simulated attention distribution;

[0010] The student branch in the vision-language-action model is trained according to the semantic consistency loss function and the attention consistency loss function. The test harassed image and the test language command are input into the trained student branch, and the control command is output to control the target robot.

[0011] Optionally, the robot control method based on semantic embedding and attention dual consistency, wherein acquiring the natural language instructions input by the user and acquiring multiple observation images collected by the target robot based on the natural language instructions specifically includes:

[0012] Obtain natural language commands input by the user and input the natural language commands into the target robot to control the movement of the target robot;

[0013] The target robot acquires observation images at multiple time points using an observation camera during its movement.

[0014] Modeling the natural language commands and each observed image yields the action sequence of the target robot:

[0015] ;

[0016] in, Indicates time points from 1 to T Action sequence, The model parameters are: The visual-language-action model, Indicates time points from 1 to T The observed images, Represents natural language instructions.

[0017] Optionally, the robot control method based on semantic embedding and attention dual consistency, wherein constructing a vision-language-action model, inputting all observed images and natural language commands into the vision-language-action model, and outputting a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution, specifically includes:

[0018] A visual-language-action model is constructed in parallel with teacher and student branches. All observed images and natural language instructions are input into the teacher branch, and the output is a true multimodal semantic representation and a true attention distribution.

[0019] After patching all the observed images, they are input into the student branch, which outputs a simulated multimodal semantic representation and a simulated attention distribution.

[0020] Optionally, the robot control method based on semantic embedding and attentional consistency, wherein the step of inputting all the observed images and the natural language commands into the teacher branch and outputting a true multimodal semantic representation and a true attention distribution specifically includes:

[0021] Freeze all parameters of the teacher branch, and input all the observed images and the natural language instructions into the teacher branch;

[0022] The teacher branch uses a visual encoder to encode features of the natural language instructions and all the observed images, and outputs a true multimodal semantic representation;

[0023] Extract the key layer set from the vision-language-action model, wherein the key layer set represents the key layers that align the natural language instruction with all observed image regions;

[0024] The natural language instruction is defined as the true query vector, and all the observed image regions are defined as true key vectors. The true attention probability distribution is extracted in each key layer based on the true query vector and the true key vector, and a true attention probability matrix is ​​constructed based on all the true attention probability distributions.

[0025] Optionally, the robot control method based on semantic embedding and attention dual consistency, wherein the step of patching all the observed images and inputting them into the student branch to output simulated multimodal semantic representation and simulated attention distribution specifically includes:

[0026] A randomly transformed universal patch is superimposed on the observation image corresponding to each time point. After performing a random geometric transformation on each universal patch, the corresponding patch image is obtained:

[0027] ;

[0028] in, Indicates the first t A patch image, A binary mask representing the patch location on a patched image. This represents element-wise multiplication. Indicates patch P Geometric transformation operators, Indicates the first t One observation image, Operations representing random geometric transformations;

[0029] Input all the patch images and the natural language instructions into the student branch, and output a simulated multimodal high-dimensional representation;

[0030] The natural language instruction is defined as a simulated query vector, all patch image regions are defined as simulated key-value vectors, simulated attention probability distributions are extracted in each key layer based on the simulated query vector and the simulated key-value vector, and a simulated attention probability matrix is ​​constructed based on all the simulated attention probability distributions.

[0031] Optionally, the robot control method based on semantic embedding and attentional consistency, wherein constructing a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and constructing an attention consistency loss function using the real attention distribution and the simulated attention distribution, specifically includes:

[0032] Calculate the consistency between real multimodal semantic representations and simulated multimodal semantic representations to construct a semantic consistency loss function:

[0033] ;

[0034] in, This represents the semantic consistency loss function. Represents true multimodal semantic representation. This represents a simulated multimodal semantic representation. express and The L2 norm between them;

[0035] Calculate the distribution difference between the real attention probability matrix and the simulated attention probability matrix to construct the attention consistency loss function:

[0036] ;

[0037] in, Represents the attention consistency loss function. Represents the set of key layers. KL divergence is used to describe the difference between two probability distributions. Indicates the first l The true attention probability distribution of each key layer Indicates the first l Simulated attention probability distribution of each key layer express and The norm of .

[0038] Optionally, the robot control method based on semantic embedding and attention consistency, wherein training the student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and inputting the test harassed image and test language command into the trained student branch, and outputting control commands to control the target robot, specifically includes:

[0039] Construct corresponding balance coefficients for the semantic consistency loss function and the attention consistency loss function respectively, and then perform a weighted fusion of the semantic consistency loss function and the attention consistency loss function based on all the balance coefficients to obtain the total loss function:

[0040] ;

[0041] in, Represents the total loss function. express The balance coefficient, express The balance coefficient, This represents the semantic consistency loss function. Represents the attention consistency loss function;

[0042] Set the visual encoder in the student branch to a trainable state, and freeze all language model parameters and action decoder parameters in the student branch;

[0043] The total loss function is used to backpropagate the student branch to update the visual encoder of the student branch;

[0044] The system acquires the user-inputted test language commands and test disturbance images, inputs the test language commands and test disturbance images into the trained student branch, and outputs robot control commands.

[0045] The target robot is controlled according to the robot control instructions, and the current action sequence of the target robot is collected.

[0046] Furthermore, to achieve the above objectives, the present invention also provides a robot control system based on semantic embedding and attentional consistency, wherein the robot control system based on semantic embedding and attentional consistency includes:

[0047] The data acquisition module is used to acquire natural language commands input by the user and to acquire multiple observation images collected by the target robot based on the natural language commands.

[0048] The teacher model output module is used to construct a vision-language-action model, inputting all the observed images and the natural language instructions into the vision-language-action model, and outputting a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution;

[0049] The data preprocessing module is used to construct a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and to construct an attention consistency loss function using the real attention distribution and the simulated attention distribution;

[0050] The model training module is used to train the student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and input the test harassed image and test language command into the trained student branch, and output control commands to control the target robot.

[0051] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a robot control program based on semantic embedding and attentional consistency stored in the memory and executable on the processor, wherein when the robot control program based on semantic embedding and attentional consistency is executed by the processor, it implements the steps of the robot control method based on semantic embedding and attentional consistency as described above.

[0052] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot control program based on semantic embedding and attentional consistency, and the robot control program based on semantic embedding and attentional consistency, when executed by a processor, implements the steps of the robot control method based on semantic embedding and attentional consistency as described above.

[0053] In this invention, natural language commands input by the user are acquired, along with multiple observation images collected by the target robot based on these commands. A vision-language-action model is constructed, and all the observation images and natural language commands are input into the model, outputting a true multimodal semantic representation, a true attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution. A semantic consistency loss function is constructed using the true and simulated multimodal semantic representations, and an attention consistency loss function is constructed using the true and simulated attention distributions. The student branch in the vision-language-action model is trained based on the semantic and attention consistency loss functions, and a test perturbation image and test language commands are input into the trained student branch, outputting control commands to control the target robot. This invention, through explicit constraints of attention consistency and patch suppression, enables cross-modal attention to converge from the patch region back to the task-related true target region, reducing the interference of adversarial patches on the model's attention logic. The lightweight fine-tuning strategy significantly improves model robustness and increases the success rate of task execution. Attached Figure Description

[0054] Figure 1 This is a flowchart of a preferred embodiment of the robot control method based on semantic embedding and attentional consistency of the present invention;

[0055] Figure 2 This is a flowchart of a preferred embodiment of the robot control method based on semantic embedding and attention consistency of the present invention.

[0056] Figure 3 This is a structural diagram of a preferred embodiment of the robot control system based on semantic embedding and attention consistency of the present invention;

[0057] Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0059] Embodied intelligence VLA models (Vision-Language-Action Models) jointly model visual input and natural language commands, directly outputting robot action sequences. However, the internal decision-making logic of existing VLA models can be hijacked by adversarial patches, causing cross-modal attention heatmaps to continuously concentrate on patch regions, leading to trajectory deviations or misgrasping. Therefore, in the embodiments disclosed in this invention, a semantic-attention aligned robust learning framework (SEAL) for embodied intelligence VLA models is adopted to resist cross-modal attention hijacking caused by physically deployable adversarial patches. This framework employs a teacher-student dual-branch training structure and explicitly distills clean-condition multimodal semantic representations and Cross-Attention (the key layer responsible for aligning language commands with image regions) structures during the training phase.

[0060] The preferred embodiment of the robot control method based on semantic embedding and attention-based dual consistency described in this invention, such as... Figure 1 As shown, the robot control method based on semantic embedding and attention-based dual consistency includes the following steps:

[0061] Step S10: Obtain the natural language command input by the user, and obtain multiple observation images collected by the target robot based on the natural language command.

[0062] Specifically, for the vision-language-action (VLA) model, robust training and deployment under physically deployable perturbations such as adversarial patches are first performed, and the collected data is then visualized and modeled.

[0063] Specifically, the natural language commands input by the user are obtained and input into the target robot to control the movement of the target robot;

[0064] The target robot acquires observation images at multiple time points using an observation camera during its movement.

[0065] Modeling the natural language commands and each observed image yields the action sequence of the target robot:

[0066] ;

[0067] in, Indicates time points from 1 to T Action sequence, The model parameters are: The visual-language-action model, Indicates time points from 1 to T The observed images, Represents natural language instructions.

[0068] Among them, such as Figure 2 As shown, firstly, natural language commands are used to control the target robot's actions. Then, based on this action process, a series of observation images are captured using a camera on the target robot. Finally, a VLA model is used to visualize the natural language commands and this series of observation images (where, Figure 2 EDPA in the text represents cutting-edge research on visual attacks against VLA models. The method involves introducing a specific loss function to update the patches attached to the visual input image of the VLA model, thereby disrupting the visual embedding output by the VLA visual encoder when the input is attacked by adversarial patches, and the visual embedding output when the input image is clean, which significantly reduces the success rate of the model in performing the task. This allows the robot's action sequence to be obtained.

[0069] Step S20: Construct a vision-language-action model, input all the observed images and the natural language commands into the vision-language-action model, and output the real multimodal semantic representation, the real attention distribution, the simulated multimodal semantic representation, and the simulated attention distribution.

[0070] In the embodiments disclosed in this invention, by constructing a teacher-student dual-branch architecture, cross-modal semantic consistency constraints and attention distribution consistency constraints are introduced during the training process, thereby improving the robustness of the model under physical world interference without significantly increasing computational costs.

[0071] Specifically, a visual-language-action model is constructed in parallel with teacher and student branches. All observed images and natural language instructions are input into the teacher branch, and the output is a real multimodal semantic representation and a real attention distribution.

[0072] After patching all the observed images, they are input into the student branch, which outputs a simulated multimodal semantic representation and a simulated attention distribution.

[0073] The method involves constructing a parallel training network with teacher and student branches, both of which are loaded with pre-trained visual-language-action model parameters. The teacher branch takes a clean image as input and freezes it completely. The student branch takes a perturbed image with adversarial patches and geometric transformations as input and learns to recover the internal representation and alignment logic of the VLA under the same language instructions. The output of the teacher branch (clean input) is frozen as a soft label, and the student branch (perturbed input) is supervised to recover the internal representation and alignment structure under the same language instructions. After training, the teacher branch is discarded, and only the student branch is retained for inference deployment.

[0074] Furthermore, all parameters of the teacher branch are frozen, and all the observed images and the natural language instructions are input into the teacher branch;

[0075] The teacher branch uses a visual encoder to encode features of the natural language instructions and all the observed images, and outputs a true multimodal semantic representation;

[0076] Extract the key layer set from the vision-language-action model, wherein the key layer set represents the key layers that align the natural language instruction with all observed image regions;

[0077] The natural language instruction is defined as the real query vector, and all the observed image regions are defined as real key vectors. The real attention probability distribution is extracted in each key layer according to the real query vector and the real key vector, and the real attention probability matrix is ​​constructed according to all the real attention probability distributions.

[0078] The real multimodal high-dimensional representation and the real attention probability matrix are both generated by the teacher branch based on all the observed images and the natural language instructions without patching.

[0079] In this process, the modeled action sequence is input into the teacher branch. Throughout the training process, all parameters of the teacher branch remain completely frozen and do not participate in gradient updates. Its role is to act as an "expert model" to provide the student branch with standard semantic representations and correct attention distribution ground values ​​under clean input.

[0080] In the embodiments disclosed in this invention, during the training process, a set of key layers (Cross-Attention) responsible for aligning language instructions with image regions is selected from the VLA model. Then, the attention probability distribution matrix of the teacher branch is extracted and head averaging is performed. Multiple attention heads are used in parallel, each independently focusing on different features or positional relationships of the input sequence. The outputs of multiple heads are directly averaged to improve the model's stability and expressive power, resulting in an attention probability map of language instructions and image regions aligned. This map is used to measure the difference in attention distribution between the two, thereby forcing the student branch to maintain the "object-instruction alignment" attention logic consistent with the clean condition even under patched perturbation conditions.

[0081] Furthermore, a randomly transformed general patch is superimposed on the observation image corresponding to each time point. After performing a random geometric transformation on each general patch, the corresponding patch image is obtained:

[0082] ;

[0083] in, Indicates the first t A patch image, A binary mask representing the patch location on a patched image. This represents element-wise multiplication. Indicates patch P Geometric transformation operators, Indicates the first t One observation image, Operations representing random geometric transformations;

[0084] Input all the patch images and the natural language instructions into the student branch, and output a simulated multimodal high-dimensional representation;

[0085] The natural language instruction is defined as a simulated query vector, all patch image regions are defined as simulated key-value vectors, simulated attention probability distributions are extracted in each key layer based on the simulated query vector and the simulated key-value vector, and a simulated attention probability matrix is ​​constructed based on all the simulated attention probability distributions.

[0086] The simulated multimodal high-dimensional representation and the simulated attention probability matrix are both generated by the student branch based on all the patched images and the natural language instructions after patching.

[0087] In order to simulate malicious recognition interference in the physical world, a general patch with geometric transformation is superimposed on each observed image. The patch can be pasted onto the image after undergoing geometric transformation, while the language instructions and model parameters remain unchanged. Then, it is forward propagated through the student branch. After undergoing the same processing as the teacher branch, the simulated multimodal semantic representation and simulated attention probability matrix corresponding to the output of the teacher branch are obtained.

[0088] Step S30: Construct a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and construct an attention consistency loss function using the real attention distribution and the simulated attention distribution.

[0089] In the model training process, visual information (clean image and patch image) and text information (task instructions) are input into the teacher branch (input clean image) and the student branch (input patch image). Then, the semantic consistency loss function and the attention consistency loss function are constructed based on the output of both. The semantic consistency loss function can force the visual encoder to ignore the pixel-level interference of the patch and map the adversarial sample to the same semantic cluster as the clean sample. The attention consistency loss function can effectively solve the problem that adversarial patches can easily cause the model's attention to shift from the target object to the patch region.

[0090] Specifically, the consistency between real multimodal semantic representations and simulated multimodal semantic representations is calculated to construct a semantic consistency loss function:

[0091] ;

[0092] in, This represents the semantic consistency loss function. Represents true multimodal semantic representation. This represents a simulated multimodal semantic representation. express and The L2 norm between them;

[0093] Calculate the distribution difference between the real attention probability matrix and the simulated attention probability matrix to construct the attention consistency loss function:

[0094] ;

[0095] in, Represents the attention consistency loss function. Represents the set of key layers. KL divergence is used to describe the difference between two probability distributions. Indicates the first l The true attention probability distribution of each key layer Indicates the first l Simulated attention probability distribution of each key layer express and The norm of .

[0096] Among them, the attention consistency loss uses the attention map of clean samples as a "soft label" to guide the perturbed model to restore the correct focus of attention. When the attention consistency loss is minimized, the model can still maintain the same "attention logic" as in the clean state even in the presence of adversarial patches. Although the key layer parameters are fixed, the attention distribution is jointly determined by text tokens (the smallest basic unit processed by the large model) (simulated query vector) and visual tokens (simulated key-value vector).

[0097] By fine-tuning the visual encoder to make the visual tokens under patched conditions closer to clean semantics (i.e., closer to the unpatched observed image), the attention structure of the key layer set is indirectly recovered, effectively resisting attention hijacking by strong visual attacks such as adversarial patching. However, for VLA models with a huge number of parameters, full fine-tuning is too costly and may lead to catastrophic forgetting. Therefore, in the embodiments disclosed in this invention, a lightweight fine-tuning strategy (fine-tuning only the visual encoder) is adopted and the VLM backbone parameters are fixed, thereby training the student branch.

[0098] Step S40: Train the student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and input the test harassed image and test language command into the trained student branch, and output control command to control the target robot.

[0099] Among them, such as Figure 2 As shown, this invention simultaneously employs a semantic consistency loss function and an attention consistency loss function to train the student branch, which can effectively constrain the consistency of multimodal semantic embedding (solving perceptual drift) and the consistency of cross-modal attention distribution (solving attention hijacking), forming a dual consistency joint loss objective, which is used to restore the stability of executable action decisions under patch perturbation conditions.

[0100] Specifically, corresponding balance coefficients are constructed for the semantic consistency loss function and the attention consistency loss function, respectively, and the semantic consistency loss function and the attention consistency loss function are weighted and fused based on all the balance coefficients to obtain the total loss function:

[0101] ;

[0102] in, Represents the total loss function. express The balance coefficient, express The balance coefficient, This represents the semantic consistency loss function. Represents the attention consistency loss function;

[0103] Set the visual encoder in the student branch to a trainable state, and freeze all language model parameters and action decoder parameters in the student branch;

[0104] The total loss function is used to backpropagate the student branch to update the visual encoder of the student branch;

[0105] The system acquires the user-inputted test language commands and test disturbance images, inputs the test language commands and test disturbance images into the trained student branch, and outputs robot control commands.

[0106] The target robot is controlled according to the robot control instructions, and the current action sequence of the target robot is collected.

[0107] In the embodiments disclosed in this invention, a multi-task joint optimization strategy is adopted, and corresponding balance coefficients are constructed for the semantic consistency loss function and the attention consistency loss function to balance task performance and robustness. The teacher branch is completely frozen, and the language and multimodal backbone parameters of the student branch are preferably frozen. During training, backpropagation is used to update only the visual encoder parameters of the student branch. This ensures that the fine-tuned visual encoder can still output feature embeddings consistent with clean visual input and maintain the consistency of the attention heatmap with clean visual input even under visual perturbation attacks. Furthermore, fine-tuning only the visual encoder without updating the backbone parameters of the VLA model helps to keep the model's task performance unaffected under clean visual input.

[0108] Furthermore, after training, the teacher branch is discarded, and only the student branch is retained for inference. During inference, the input is the test perturbation image and the instruction, and the output is the action sequence. It should be noted that the test perturbation image includes images with adversarial patches and physical perturbations, not just partially missing images. No additional branches or detection modules are required, and the deployment overhead is not increased.

[0109] This invention uses explicit constraints of attention consistency and patch suppression to enable cross-modal attention to converge back from the patch region to the task-related real target region, reducing the interference of adversarial patches on the model's attention logic. The lightweight fine-tuning strategy significantly improves the model's robustness and increases the success rate of task execution.

[0110] Furthermore, such as Figure 3 As shown, based on the above-described robot control method based on semantic embedding and attentional consistency, this invention also provides a robot control system based on semantic embedding and attentional consistency, wherein the robot control system based on semantic embedding and attentional consistency includes:

[0111] The data acquisition module 51 is used to acquire natural language commands input by the user and acquire multiple observation images collected by the target robot based on the natural language commands.

[0112] The teacher model output module 52 is used to construct a vision-language-action model, inputting all the observed images and the natural language instructions into the vision-language-action model, and outputting a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution;

[0113] Data preprocessing module 53 is used to construct a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and to construct an attention consistency loss function using the real attention distribution and the simulated attention distribution;

[0114] The model training module 54 is used to train the student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and input the test harassed image and test language command into the trained student branch, and output control commands to control the target robot.

[0115] Furthermore, such as Figure 4 As shown, based on the above-mentioned robot control method and system based on semantic embedding and attention dual consistency, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0116] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage units. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a robot control program 40 based on semantic embedding and attentional consistency, which can be executed by the processor 10 to implement the robot control method based on semantic embedding and attentional consistency in this application.

[0117] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the robot control method based on semantic embedding and attention dual consistency.

[0118] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0119] In one embodiment, when processor 10 executes robot control program 40 based on semantic embedding and attention-based dual consistency in memory 20, the following steps are performed:

[0120] Acquire natural language commands input by the user, and acquire multiple observation images collected by the target robot based on the natural language commands;

[0121] A vision-language-action model is constructed, and all the observed images and natural language commands are input into the vision-language-action model. The model outputs a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution.

[0122] A semantic consistency loss function is constructed using the real multimodal semantic representation and the simulated multimodal semantic representation, and an attention consistency loss function is constructed using the real attention distribution and the simulated attention distribution;

[0123] The student branch in the vision-language-action model is trained according to the semantic consistency loss function and the attention consistency loss function. The test harassed image and the test language command are input into the trained student branch, and the control command is output to control the target robot.

[0124] The step of acquiring natural language commands input by the user and acquiring multiple observation images collected by the target robot based on the natural language commands specifically includes:

[0125] Obtain natural language commands input by the user and input the natural language commands into the target robot to control the movement of the target robot;

[0126] The target robot acquires observation images at multiple time points using an observation camera during its movement.

[0127] Modeling the natural language commands and each observed image yields the action sequence of the target robot:

[0128] ;

[0129] in, Indicates time points from 1 to T Action sequence, The model parameters are: The visual-language-action model, Indicates time points from 1 to T The observed images, Represents natural language instructions.

[0130] The construction of the vision-language-action model involves inputting all observed images and natural language commands into the model and outputting a true multimodal semantic representation, a true attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution. Specifically, this includes:

[0131] A visual-language-action model is constructed in parallel with teacher and student branches. All observed images and natural language instructions are input into the teacher branch, and the output is a true multimodal semantic representation and a true attention distribution.

[0132] After patching all the observed images, they are input into the student branch, which outputs a simulated multimodal semantic representation and a simulated attention distribution.

[0133] Specifically, the step of inputting all the observed images and the natural language instructions into the teacher branch and outputting a true multimodal semantic representation and a true attention distribution includes:

[0134] Freeze all parameters of the teacher branch, and input all the observed images and the natural language instructions into the teacher branch;

[0135] The teacher branch uses a visual encoder to encode features of the natural language instructions and all the observed images, and outputs a true multimodal semantic representation;

[0136] Extract the key layer set from the vision-language-action model, wherein the key layer set represents the key layers that align the natural language instruction with all observed image regions;

[0137] The natural language instruction is defined as the true query vector, and all the observed image regions are defined as true key vectors. The true attention probability distribution is extracted in each key layer based on the true query vector and the true key vector, and a true attention probability matrix is ​​constructed based on all the true attention probability distributions.

[0138] Specifically, the process of patching all observed images and inputting them into the student branch to output simulated multimodal semantic representations and simulated attention distributions includes:

[0139] A randomly transformed universal patch is superimposed on the observation image corresponding to each time point. After performing a random geometric transformation on each universal patch, the corresponding patch image is obtained:

[0140] ;

[0141] in, Indicates the first t A patch image, A binary mask representing the patch location on a patched image. This represents element-wise multiplication. Indicates patch P Geometric transformation operators, Indicates the first t One observation image, Operations representing random geometric transformations;

[0142] Input all the patch images and the natural language instructions into the student branch, and output a simulated multimodal high-dimensional representation;

[0143] The natural language instruction is defined as a simulated query vector, all patch image regions are defined as simulated key-value vectors, simulated attention probability distributions are extracted in each key layer based on the simulated query vector and the simulated key-value vector, and a simulated attention probability matrix is ​​constructed based on all the simulated attention probability distributions.

[0144] Specifically, the step of constructing a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and constructing an attention consistency loss function using the real attention distribution and the simulated attention distribution, includes:

[0145] Calculate the consistency between real multimodal semantic representations and simulated multimodal semantic representations to construct a semantic consistency loss function:

[0146] ;

[0147] in, Represents the semantic consistency loss function. Represents true multimodal semantic representation. This represents a simulated multimodal semantic representation. express and The L2 norm between them;

[0148] Calculate the distribution difference between the real attention probability matrix and the simulated attention probability matrix to construct the attention consistency loss function:

[0149] ;

[0150] in, Represents the attention consistency loss function. Represents the set of key layers. KL divergence is used to describe the difference between two probability distributions. Indicates the first l The true attention probability distribution of each key layer Indicates the first l Simulated attention probability distribution of each key layer express and The norm of .

[0151] Specifically, the step of training the student branch in the vision-language-action model based on the semantic consistency loss function and the attention consistency loss function, inputting the test harassed image and test language command into the trained student branch, and outputting control commands to control the target robot includes:

[0152] Construct corresponding balance coefficients for the semantic consistency loss function and the attention consistency loss function respectively, and then perform a weighted fusion of the semantic consistency loss function and the attention consistency loss function based on all the balance coefficients to obtain the total loss function:

[0153] ;

[0154] in, Represents the total loss function. express The balance coefficient, express The balance coefficient, Represents the semantic consistency loss function. Represents the attention consistency loss function;

[0155] Set the visual encoder in the student branch to a trainable state, and freeze all language model parameters and action decoder parameters in the student branch;

[0156] The total loss function is used to backpropagate the student branch to update the visual encoder of the student branch;

[0157] The system acquires the user-inputted test language commands and test disturbance images, inputs the test language commands and test disturbance images into the trained student branch, and outputs robot control commands.

[0158] The target robot is controlled according to the robot control instructions, and the current action sequence of the target robot is collected.

[0159] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot control program based on semantic embedding and attentional consistency, and the robot control program based on semantic embedding and attentional consistency, when executed by a processor, implements the steps of the robot control method based on semantic embedding and attentional consistency as described above.

[0160] In summary, this invention provides a robot control method and related equipment based on semantic embedding and attentional consistency. The method includes: acquiring natural language commands input by a user and acquiring multiple observation images collected by the target robot based on the natural language commands; constructing a vision-language-action model, inputting all the observation images and the natural language commands into the vision-language-action model, and outputting a real multimodal semantic representation, a real attention distribution, a simulated multimodal semantic representation, and a simulated attention distribution; constructing a semantic consistency loss function using the real multimodal semantic representation and the simulated multimodal semantic representation, and constructing an attention consistency loss function using the real attention distribution and the simulated attention distribution; training a student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and inputting a test harassed image and a test language command into the trained student branch, and outputting control commands to control the target robot. This invention uses explicit constraints of attention consistency and patch suppression to enable cross-modal attention to converge back from the patch region to the task-related real target region, reducing the interference of adversarial patches on the model's attention logic. The lightweight fine-tuning strategy significantly improves the model's robustness and increases the success rate of task execution.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0162] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0163] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A robot control method based on semantic embedding and attention-based dual consistency, characterized in that, The robot control method based on semantic embedding and attention-based dual consistency includes: Acquire natural language commands input by the user, and acquire multiple observation images collected by the target robot based on the natural language commands; A vision-language-action model is constructed, and all the observed images and natural language commands are input into the vision-language-action model. The model outputs a real multimodal high-dimensional representation, a real attention probability matrix, a simulated multimodal high-dimensional representation, and a simulated attention probability matrix. The construction of the vision-language-action model involves inputting all observed images and natural language commands into the model and outputting a real multimodal high-dimensional representation, a real attention probability matrix, a simulated multimodal high-dimensional representation, and a simulated attention probability matrix. Specifically, this includes: A visual-language-action model is constructed in parallel with teacher and student branches. All observed images and natural language commands are input into the teacher branch, and the output is a real multimodal high-dimensional representation and a real attention probability matrix. After patching all the observed images, they are input into the student branch, and the output is a simulated multimodal high-dimensional representation and a simulated attention probability matrix; A semantic consistency loss function is constructed using the real multimodal high-dimensional representation and the simulated multimodal high-dimensional representation, and an attention consistency loss function is constructed using the real attention probability matrix and the simulated attention probability matrix; The student branch in the vision-language-action model is trained according to the semantic consistency loss function and the attention consistency loss function. The test harassed image and the test language command are input into the trained student branch, and the control command is output to control the target robot.

2. The robot control method based on semantic embedding and attention-based dual consistency according to claim 1, characterized in that, The process of acquiring natural language commands input by the user and acquiring multiple observation images collected by the target robot based on the natural language commands specifically includes: Obtain natural language commands input by the user and input the natural language commands into the target robot to control the movement of the target robot; The target robot acquires observation images at multiple time points using an observation camera during its movement. Modeling the natural language commands and each observed image yields the action sequence of the target robot: ； in, Indicates time points from 1 to T Action sequence, The model parameters are: The visual-language-action model, Indicates time points from 1 to T The observed images, Represents natural language instructions.

3. The robot control method based on semantic embedding and attention-based dual consistency according to claim 1, characterized in that, The process of inputting all the observed images and the natural language instructions into the teacher branch and outputting a true multimodal high-dimensional representation and a true attention probability matrix specifically includes: All parameters of the teacher branch are frozen, and all the observed images and natural language instructions are input into the teacher branch to output a true multimodal high-dimensional representation. Extract the key layer set from the vision-language-action model, wherein the key layer set represents the key layers that align the natural language instruction with all observed image regions; The natural language instruction is defined as the true query vector, and all the observed image regions are defined as true key vectors. The true attention probability distribution is extracted in each key layer based on the true query vector and the true key vector, and a true attention probability matrix is ​​constructed based on all the true attention probability distributions.

4. The robot control method based on semantic embedding and attentional consistency according to claim 1, characterized in that, The process of patching all the observed images and then inputting them into the student branch outputs a simulated multimodal high-dimensional representation and a simulated attention probability matrix, specifically including: A randomly transformed general patch is superimposed on the observed image at each time point. After performing a random geometric transformation on each general patch, the corresponding patch image is obtained: ； in, Indicates the first t A patch image, A binary mask representing the patch location on a patched image. This represents element-wise multiplication. Describes the geometric transformation operator for patch P. Indicates the first t One observation image, Operations representing random geometric transformations; Input all the patch images and the natural language instructions into the student branch, and output a simulated multimodal high-dimensional representation; The natural language instruction is defined as a simulated query vector, all patch image regions are defined as simulated key-value vectors, simulated attention probability distributions are extracted in each key layer based on the simulated query vector and the simulated key-value vector, and a simulated attention probability matrix is ​​constructed based on all the simulated attention probability distributions.

5. The robot control method based on semantic embedding and attention-based dual consistency according to claim 1, characterized in that, The step of constructing a semantic consistency loss function using the real multimodal high-dimensional representation and the simulated multimodal high-dimensional representation, and constructing an attention consistency loss function using the real attention probability matrix and the simulated attention probability matrix, specifically includes: Calculate the consistency between real multimodal high-dimensional representations and simulated multimodal high-dimensional representations to construct a semantic consistency loss function: ； in, Represents the semantic consistency loss function. Represents a true multimodal high-dimensional representation. This represents a high-dimensional representation of multimodal simulation. express and The L2 norm between them; Calculate the distribution difference between the real attention probability matrix and the simulated attention probability matrix to construct the attention consistency loss function: ； in, Represents the attention consistency loss function. Represents the set of key layers. KL divergence is used to describe the difference between two probability distributions. Indicates the first l The true attention probability distribution of each key layer Indicates the first l Simulated attention probability distribution of each key layer express and The norm of .

6. The robot control method based on semantic embedding and attentional consistency according to claim 1, characterized in that, The process of training the student branch in the vision-language-action model based on the semantic consistency loss function and the attention consistency loss function, inputting the test harassed image and test language command into the trained student branch, and outputting control commands to control the target robot specifically includes: Construct corresponding balance coefficients for the semantic consistency loss function and the attention consistency loss function respectively, and then perform a weighted fusion of the semantic consistency loss function and the attention consistency loss function based on all the balance coefficients to obtain the total loss function: ； in, Represents the total loss function. express The balance coefficient, express The balance coefficient, Represents the semantic consistency loss function. Represents the attention consistency loss function; Set the visual encoder in the student branch to a trainable state, and freeze all language model parameters and action decoder parameters in the student branch; The total loss function is used to backpropagate the student branch to update the visual encoder of the student branch; The system acquires the user-inputted test language commands and test disturbance images, inputs the test language commands and test disturbance images into the trained student branch, and outputs robot control commands. The target robot is controlled according to the robot control instructions, and the current action sequence of the target robot is collected.

7. A robot control system based on semantic embedding and attentional consistency, characterized in that, The robot control system based on semantic embedding and attentional consistency is used to implement the robot control method based on semantic embedding and attentional consistency as described in any one of claims 1-6, wherein the robot control system based on semantic embedding and attentional consistency includes: The data acquisition module is used to acquire natural language commands input by the user and to acquire multiple observation images collected by the target robot based on the natural language commands. The teacher model output module is used to construct a vision-language-action model. It inputs all the observed images and the natural language instructions into the vision-language-action model and outputs a real multimodal high-dimensional representation, a real attention probability matrix, a simulated multimodal high-dimensional representation, and a simulated attention probability matrix. The data preprocessing module is used to construct a semantic consistency loss function using the real multimodal high-dimensional representation and the simulated multimodal high-dimensional representation, and to construct an attention consistency loss function using the real attention probability matrix and the simulated attention probability matrix; The model training module is used to train the student branch in the vision-language-action model according to the semantic consistency loss function and the attention consistency loss function, and input the test harassed image and test language command into the trained student branch, and output control commands to control the target robot.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a robot control program based on semantic embedding and attentional consistency stored in the memory and executable on the processor. When the robot control program based on semantic embedding and attentional consistency is executed by the processor, it implements the steps of the robot control method based on semantic embedding and attentional consistency as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a robot control program based on semantic embedding and attentional consistency, which, when executed by a processor, implements the steps of the robot control method based on semantic embedding and attentional consistency as described in any one of claims 1-6.

Images