Multi-object oriented motion sequence generation system, method, and robot control method

Abstract

The application discloses a multi-object-oriented action sequence generation system and method and a robot control method, relates to the technical field of artificial intelligence, and solves the technical problems of target selection deviation and operation failure caused by perception attention drift and strategy lazy learning in a multi-object scene.The technical solution points are that a perception grounding expert module performs explicit space coordinate regression on the extracted fusion feature map, and outputs the normalized boundary box coordinates of the target object;then the coordinates are encoded into independent anchor token, and a decoupled input sequence of visual token, state token, anchor token and action token is constructed;the decoupled input sequence is modulated, and finally the anchor token is forced to be queried by the action expert module in each denoising step, so that a short-time action block sequence of the precise anchor target is generated and delivered to the robot executor to drive the current operation step.The target selection accuracy in the multi-object interference task is improved, and the instruction following robustness and operation success rate are enhanced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a system, method and robot control method for generating action sequences for multiple objects. Background Technology

[0002] Vision-Language-Action (VLA) models are a core technology in the field of embodied intelligence, aiming to extend the general understanding capabilities of large-scale Visual Language Models (VLMs) to robot control tasks. Existing VLA architectures, such as RT-1, OpenVLA, and the π0 model with a diffusion strategy, such as... Figure 1 As shown, a pre-trained VLM model is typically used to extract the fusion features of environmental images and text instructions, and these features are used as constraints for generating robot actions.

[0003] However, existing technologies reveal several core shortcomings when dealing with complex operational scenarios involving multiple interfering objects: (1) "Attention drift" at the perception level leads to target recognition bias: In environments with multiple objects, existing technologies often fail to accurately map the semantic descriptions in language instructions (such as "red bell") to the correct visual references in physical space. Existing models typically only apply supervision at the task level, i.e., only optimizing the accuracy of action generation, while ignoring explicit guidance on "where to look". This indirect supervision mechanism is prone to triggering "shortcut learning" at the perception level, causing cross-modal attention within the model to drift when there are many distractions, mistakenly pointing instructions to non-target objects. Robots often exhibit the phenomenon of "calling a deer a horse", incorrectly applying operation instructions to distractions in the environment.

[0004] (2) "Lazy learning" at the policy level leads to the shielding of perceptual information: Research has found that even if the model's perceptual alignment is corrected through auxiliary means, the existing policy generation mechanism still cannot guarantee the correct execution of actions. The lack of a strong explicit coupling mechanism between the perception layer and the action generation layer in the existing architecture leads to the defect of "lazy learning" in the action decoder. Even if the perception layer provides accurate object localization, the action generation module will actively ignore these high-dimensional perceptual signals in order to minimize training loss, and instead anchor the prediction to static or irrelevant areas of the image. This disconnect between perception and action means that even if the robot sees the target correctly, the final generated control trajectory still cannot be accurately applied to the target object, resulting in task failure.

[0005] (3) Insufficient robustness of command following due to lack of multi-level grounding constraints: Existing technologies typically treat positioning and action generation as a black box process, lacking hierarchical explicit constraints. When processing fine-grained commands such as pressing the top of an alarm clock with a transparent screen, a single action loss function is insufficient to simultaneously handle complex semantic parsing, precise spatial positioning, and coherent physical control. In scenarios with multiple object interference, the accuracy of command following drops significantly, and the system struggles to maintain stable operational precision in dynamic and changing environments. Summary of the Invention

[0006] This application provides a system, method, and robot control method for generating action sequences for multiple objects. The technical objective is to improve the accuracy of action sequence generation in complex environments, thereby enhancing the robot's target selection accuracy and operation success rate.

[0007] The above-mentioned technical objective of this application is achieved through the following technical solution: A multi-object action sequence generation system includes: The data acquisition module acquires visual observation images of the robot's operation scene, natural language text commands, and the robot arm's body state information. A visual language perception backbone network obtains a fused visual feature map based on the visual observation image and the natural language text instruction, and obtains a visual token based on the fused visual feature map; The sensing grounding expert module obtains the normalized bounding box coordinates of the target object in the visual observation image based on the fused visual feature map; The decoupling guidance module encodes and projects the normalized bounding box coordinates to obtain anchor tokens, and constructs a decoupling input sequence based on the order of the visual token, state token, anchor token, and action token; wherein, the state token is obtained through the robot's body state information; during the training phase, the action token is composed of real action blocks with added noise, and during the inference phase, the action token is composed of random noise; The modulation module modulates the attention score of the visual token in the decoupled input sequence to obtain the modulated decoupled input sequence; The action expert module performs conditional diffusion generation processing on the modulated and decoupled input sequence to obtain a short-time action block sequence.

[0008] Furthermore, the visual language perception backbone network includes: The trained visual language action model performs cross-modal feature fusion processing on the visual observation image and the natural language text instruction to obtain a fused visual feature map that is semantically aligned with the natural language text instruction.

[0009] Furthermore, the sensing grounding expert module includes: The convolutional head performs structured dimensionality reduction processing on the fused visual feature map to obtain a dimensionality-reduced feature vector; wherein, the convolutional head includes a 2D convolutional layer, a batch normalization layer, a ReLU activation function and a flattening layer connected in sequence; The multilayer perceptron performs coordinate regression calculation on the dimensionality-reduced feature vector to obtain the bounding box coordinates of the target object in the visual observation image, and then normalizes the bounding box coordinates to obtain normalized bounding box coordinates.

[0010] Furthermore, the loss function of the sensing grounding expert module during the training process is expressed as: ; ; in, This represents the explicit target localization loss function of the sensing grounding expert module. Indicates GIoU loss, Indicates the coordinates of the predicted target bounding box. Represents the coordinates of the actual target bounding box; This represents the learnable parameters of the sensing grounding expert module; This represents the fused visual feature map; This represents the bounding box regression function implemented by the sensing grounding expert module. It represents the four coordinate components of the target bounding box coordinates in a four-dimensional real coordinate space.

[0011] Furthermore, the decoupling guidance module includes: The encoder maps the low-dimensional normalized bounding box coordinates to a high-dimensional embedding vector in the high-dimensional latent space. The projector projects the high-dimensional embedding vector onto the lexical dimension in the action expert module to obtain anchor tokens. The sequence construction unit constructs a decoupled input sequence based on the order of the visual token, the state token, the anchor token, and the action token, as follows: ;in, Indicates a visual token. Indicates a status token. Indicates anchor token, Indicates an action token.

[0012] Furthermore, the motion expert module is a Transformer-based diffusion motion generation model, and its loss function is: ; ; in, Indicates loss of action, This represents random noise added to the action sequence. This represents the noise predicted by the motion expert module. Indicates the diffusion time step. This represents the condition context, which includes the state token and the anchor token. Uncertainty weights are used to balance the loss generated by the action; Indicates the weighting coefficient; Expressing expectations; This represents the spread time step in the short-time action block sequence output by the motion expert module. The corresponding single-step action.

[0013] Furthermore, the modulation module is specifically used to: modulate the attention score of the visual token in the decoupled input sequence to obtain a modulated decoupled input sequence; For the first The relevance scores between each visual token and the target object are calculated to obtain a set of visual token relevance scores, represented as follows: , , ; ; in, Indicates the first A relevance score between a visual token and the target object. Indicates the first A visual token This indicates that the visual token and the anchor token are concatenated using features. This represents a multilayer perceptron used to calculate the correlation score. This indicates that normalization is performed using the Sigmoid function; Represents the set of visual token relevance scores; The token-level relevance score is calculated based on the visual token relevance score set and is expressed as follows: ; in, Indicates from action token To visual token Token-level relevance score, The query vector representing the action token. Represent the key vector and value vector of a visual token; Indicates the strength of the shielding. express and The dimension; The attention score of the visual token in the decoupled input sequence is modulated based on the token-level relevance score to obtain the modulated decoupled input sequence.

[0014] Furthermore, the motion expert module is specifically used for: Actively query the anchor tokens in the modulated and decoupled input sequence to obtain target spatial positioning information; Based on the target spatial positioning information, for each diffusion time step The noise component in the current noisy action block is predicted, and represented as: ; in, This represents a noise prediction network, specifically a Transformer-based diffusion-based action generation model. This represents the predicted value from the noise prediction network; Indicates diffusion time step The corresponding action block; This represents the modulated and decoupled input sequence; For predicted values Perform stepwise denoising to obtain the previous diffusion time step. The corresponding action block is represented as: ; in, Indicates the previous diffusion time step The corresponding action block, This represents a single-step denoising function. Indicates diffusion time step The corresponding action block; After from arrive After multi-step iterative denoising, the final executable short-time action block sequence is obtained, represented as: ; in, This indicates the action block corresponding to diffusion time step 0. Indicates time A single-step action within an action block.

[0015] A method for generating action sequences for multiple objects, implemented through the aforementioned action sequence generation system, includes: The system acquires visual observation images, natural language text commands, and the body state information of the robotic arm in the robot's operation scene. A fused visual feature map is obtained based on the visual observation image and the natural language text instruction, and a visual token is obtained based on the fused visual feature map; The normalized bounding box coordinates of the target object in the visual observation image are obtained based on the fused visual feature map. The normalized bounding box coordinates are encoded and projected to obtain anchor tokens. A decoupled input sequence is constructed according to the order of the visual token, state token, anchor token, and action token. The state token is obtained based on the robot's current state. During the training phase, the action token is composed of real action blocks with added noise, and during the inference phase, the action token is composed of random noise. The attention score of the visual token in the decoupled input sequence is modulated to obtain the modulated decoupled input sequence; The modulated and decoupled input sequence is subjected to conditional diffusion generation to obtain a short-time action block sequence.

[0016] A robot control method for multiple objects controls the robot's operation through a short-time action block sequence.

[0017] The above technical solution can achieve at least some of the following technical effects: (1) It overcomes the shortcomings of composite shortcut learning in complex multi-object scenarios and achieves highly robust target selection. Existing VLA models often experience performance degradation in cluttered environments with dense targets. This application is the first to systematically diagnose that this failure is caused by the combined effect of attention drift in the perception layer and lazy learning in the policy layer. By introducing a two-layer explicit grounding mechanism, this application establishes a rigorous causal chain from semantic localization to physical execution, which significantly improves the target selection accuracy and operational reliability of the robot in environments with multiple interferences.

[0018] (2) Explicit localization at the perception layer corrects attention drift, ensuring the robot correctly identifies the target. Addressing the issue of existing end-to-end models being susceptible to interference from similar objects, this application designs a perception-grounded expert module for target localization under linguistic conditions. By employing explicit bounding box regression and strong constraints from the loss function, the model is forced to precisely focus cross-modal attention on specific target physical entities, combined with… Figure 5 (The second line only introduces the Perception Grounding Expert Module) shows that after introducing the Perception Grounding Expert Module, the attention of the visual language action model was successfully transferred from the wrong object and aligned with the correct operation target, fundamentally eliminating perceptual visual ambiguity in multi-target scenarios.

[0019] (3) Decoupling guidance at the strategy layer suppresses lazy learning and ensures that the robot's operations are aligned with the target. Addressing the problem that existing action decoders tend to ignore visual perception signals, this application innovatively proposes a decoupling guidance mechanism. By encoding the target spatial coordinates into independent anchor tokens and reconstructing the input sequence, this system forms a controlled information bottleneck in its physical architecture. For example... Figure 5 (As shown in the third line, which introduces both the perception grounding expert module and the decoupling guidance module), this forces the action expert module to actively query the anchor point in each denoising generation step. Its attention is ultimately successfully anchored to the target object, effectively preventing the lazy behavior of the strategy layer blindly anchoring to irrelevant areas.

[0020] (4) Significantly improved the success rate and target selection accuracy of complex operational tasks. For example... Figure 6 and Figure 7 As shown, through extensive simulation benchmarks (such as RoboTwin 2.0) and real-world robot platform experiments, this application demonstrates a breakthrough performance improvement compared to existing strong benchmark models. In fine manipulation tasks involving complex multiple objects, this application achieves a significant improvement of 25.0% to 40.0% in target selection and manipulation accuracy, while increasing the success rate of end-to-end long sequence tasks by 10.0% to 25.0%.

[0021] (5) Masking visual tokens with low information content or irrelevant to the target allows the motion expert module to focus more on visual regions related to the target object. In the prior art, motion tokens sometimes assign high attention to visual tokens that are irrelevant to the target or have low information content. This attention allocation can introduce incorrect visual anchors during the action generation stage, making it difficult for the motion expert module to consistently focus on the correct target in multi-object scenes, thereby exacerbating the lazy learning phenomenon at the strategy level. To reduce the interference of irrelevant visual tokens on the action generation process, this application considers masking visual tokens with low information content or irrelevant to the target, allowing the motion expert module to focus more on visual regions related to the target object. Attached Figure Description

[0022] Figure 1 This is a flowchart of the VLM model in the existing technology; Figure 2 This is a flowchart of the action sequence generation system in the embodiments of this application; Figure 3 This is a flowchart of the sensing grounding expert module in an embodiment of this application; Figure 4 This is a flowchart illustrating the decoupled guidance module and action expert module in an embodiment of this application. Figure 5This is a comparison chart showing the effectiveness of the action sequence method of this application in target selection with other methods; Figure 6 The figures shown are simulation results based on RoboTwin 2.0 in the embodiments of this application. Figure 7 The figure shows the actual experimental results based on Agilex-piper in the embodiments of this application. Detailed Implementation

[0023] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0024] like Figure 2 As shown, the action sequence generation system for multiple objects described in this application includes a data acquisition module, a visual language perception backbone network (VLM Backbone), a perception grounding expert module (PGE), a decoupling guidance module, a modulation module, and an action expert module (AE).

[0025] The data acquisition module is used to visually observe images of the robot's operating scene at the current moment. Natural language text commands and the robot arm's body status information (Such as joint angles, end effector pose, etc.) are acquired.

[0026] In one exemplary embodiment, a two-dimensional or three-dimensional image of the environment at the current moment is captured by a vision sensor deployed within the robot's workspace, such as a fixed-view global RGB-D camera or a wrist camera mounted on the end effector of the robotic arm; this is known as a visual observation image. The visual observation image typically includes both the target object to which the operational command is directed and at least one or more interfering objects in the environment that are similar to the target in appearance, shape, or semantics. This visual observation image data forms the basic visual reference for the model to understand the physical space and distinguish the objects being manipulated.

[0027] Furthermore, natural language text instructions issued by the user or the upper-level task planning system are collected. For example, the instructions may be "pick up the red bell", "press the top of the alarm clock with the transparent screen", or "put the square block into the round groove", etc., which contain semantic attribute descriptions of the operation target, the type of action to be performed, and guidance on the potential operation parts. It is a key bridge to transform vague human intentions into precise action sequences that the robot can resolve.

[0028] This application also simultaneously collects the robot arm's body state information. This process involves reading the motor encoders installed at each joint of the robot arm to obtain precise joint angle data, and combining this data with a known kinematic model to calculate the position and orientation of the end effector in Cartesian space at the current moment, i.e., the end effector pose. In an optional implementation, the body state information can also be expanded to include the angular velocities of each joint, the opening and closing width of the end effector, and readings from force-torque sensors, to obtain a complete description of the robot's real-time kinematics and dynamics.

[0029] In one exemplary embodiment, all three types of data are at the same timestamp. The data is collected in strict synchronization to ensure strict temporal alignment between the visual scene, task intent, and robot physical state. This synchronized data constitutes the sole raw input for all subsequent data processing, and its quality and completeness directly affect the success rate of the final operation.

[0030] In one exemplary embodiment, based on ontology state information This constitutes a status token.

[0031] The visual language perception backbone network is used to obtain a fused visual feature map based on the visual observation image and the natural language text instruction, and to obtain a visual token based on the fused visual feature map.

[0032] In one exemplary embodiment, the visual language perception backbone network is a trained visual language action model that processes the visually observed images. and the natural language text instructions Cross-modal feature fusion processing is performed to obtain a fused visual feature map that is semantically aligned with the natural language text instruction text.

[0033] The Perception Grounding Expert Module is used to obtain the normalized bounding box coordinates of the target object in the visual observation image based on the fused visual feature map.

[0034] Specifically, the sensing grounding expert module includes a lightweight sensing grounding head.

[0035] In one exemplary embodiment, such as Figure 3 As shown, the aforementioned sensing grounding head includes a convolution head and a multilayer sensor.

[0036] The fused visual feature map The spatial features are sequentially processed through a 2D convolutional layer, a batch normalization layer, a ReLU activation function, and a flattening layer to achieve structured dimensionality reduction, resulting in a dimensionality-reduced feature vector. The multi-channel filters in the convolutional layers can capture spatial features from different perspectives, the batch normalization layer is used to stabilize training and accelerate convergence, the ReLU activation function introduces nonlinearity, and the flattening layer stretches the two-dimensional feature map into a one-dimensional vector, preparing it for subsequent fully connected processing.

[0037] After dimensionality reduction, the feature vector is input into a multilayer perceptron for coordinate regression calculation to obtain the bounding box coordinates of the target object in the visually observed image. These bounding box coordinates are then normalized to obtain normalized bounding box coordinates, expressed as follows: ; in, Represents the normalized bounding box coordinates of the predicted target; This represents the learnable parameters of the sensing grounding expert module; This represents the fused visual feature map; This represents the bounding box regression function implemented by the perception grounding expert module, which is used to map the fused visual feature map into the spatial localization result of the target object; It represents the four coordinate components of the target bounding box coordinates in a four-dimensional real coordinate space.

[0038] In one exemplary embodiment, the multilayer perceptron includes at least one multilayer perceptron regressor. This regressor consists of several fully connected layers and is responsible for performing a nonlinear mapping task from abstract feature vectors to specific geometric coordinates; its final output layer contains four neurons, each corresponding to one of the four normalized values ​​used to describe a two-dimensional bounding box: the x-coordinate of the center point, the y-coordinate of the center point, the width of the box, and the height of the box, i.e., the four coordinate components of the target bounding box coordinates.

[0039] To simultaneously constrain action generation and target spatial localization capabilities during end-to-end training, in one exemplary embodiment, this application introduces a perceptual grounding loss and jointly optimizes it with the action generation loss. This perceptual grounding loss employs a GIoU loss to constrain the predicted target bounding box. bounding box of the real target Consistency between them is represented as: ; in, This represents the explicit target localization loss function of the sensing grounding expert module. Indicates GIoU loss, Represents the normalized bounding box coordinates of the real target.

[0040] To completely solve the problem of lazy learning where motion experts actively ignore localization information, this application designs a decoupled guidance module at the strategy layer, structurally eliminating the possibility of shortcut learning. This decoupled guidance module includes an encoder, a projector, and a sequence construction unit.

[0041] Specifically, the bounding box coordinates are output by the sensing grounding expert module. After that, do not directly use the bounding box coordinates A simple feature concatenation is performed with the robot's body state information. Conversely, the bounding box coordinates... The data is first fed into a lightweight MLP object encoder. This MLP object encoder converts the low-dimensional bounding box coordinates... The mapping is a compact embedding representation in a high-dimensional latent space, defined as an anchor token, and represented as: ; in, Indicates anchor token, This represents a high-dimensional embedding vector. This represents a projector, whose function is to project high-dimensional embedding vectors. Project the lexical dimension into the action expert module so that it can be input into the action expert as a regular lexical.

[0042] The independence of anchor tokens is crucial, indicating that the path of anchor tokens in the model is completely separated from that of state tokens and action tokens, possessing their own clear identification and functional positioning. Their role is similar to system prompts or task prefixes in large language models, setting a persistent and unavoidable constraint for the entire sequence generation process: "All generated actions must be anchored to this spatial location I refer to." The entire encoding process provides a structurally decisive prerequisite for severing lazy learning links and forcing the policy network to faithfully adhere to visual guidance.

[0043] In one exemplary embodiment, the projector employs a lightweight two-layer MLP structure, each MLP structure including a first linear mapping layer, a nonlinear activation function, a second linear mapping layer, and a normalization layer.

[0044] Obtaining Anchor Tokens Based on this, the sequence building unit constructs a specific sequence arrangement format, namely: ,in, Indicates a visual token. Indicates a status token. Indicates anchor token, This refers to an action token. During the training phase, the action token consists of real action blocks with added noise; during the inference phase, the action token consists of random noise.

[0045] like Figure 4 As shown, the anchor token represents the target location. Strictly inserted in the state token After that, action token Previously, this decoupling design physically separated "where the target is ( "What's happening with the robot?" and "What's going on with the robot?" The anchor token acts as a system prompt, similar to that in a large language model, maintaining a persistent constraint throughout the generation of action sequences.

[0046] The modulation module is used to modulate the attention score of the visual token in the decoupled input sequence to obtain the modulated decoupled input sequence.

[0047] In one exemplary embodiment, the modulation module is specifically used for: (1) For the first The relevance scores between each visual token and the target object are calculated to obtain a set of visual token relevance scores, represented as follows: , ; ; in, Indicates the first A relevance score between a visual token and the target object. The value indicates that the relevance score, after normalization, is between 0 and 1. The larger the value, the more likely the visual token is to be related to the target object; the smaller the value, the more likely it is to belong to the background or irrelevant interference. Indicates the first A visual token This indicates that the visual token and the anchor token are concatenated using features. This represents a multilayer perceptron used to calculate the correlation score. This indicates that normalization is performed using the Sigmoid function; This represents the visual token relevance score set, used to suppress irrelevant visual information during the attention computation from action token to visual token.

[0048] (2) The token-level relevance score is calculated based on the visual token relevance score set, and is expressed as: ; in, Indicates from action token To visual token Token-level relevance score, The query vector representing the action token. Represent the key vector and value vector of a visual token; express and Dimensions. Indicates the strength of cover; in the early stages of training, it can be set to... A smaller value is initially selected, and then gradually increased during the second-stage joint fine-tuning process by action experts, thereby achieving progressive action-side masking. This approach preserves the pre-trained perceptual capabilities of the visual-language action model while suppressing interference from irrelevant visual tokens during the action generation stage.

[0049] (3) Modulate the attention score of the visual token in the decoupled input sequence according to the token-level relevance score, so as to reduce the attention weight of the target-irrelevant visual token in the action generation stage, and obtain the modulated decoupled input sequence.

[0050] The modulated and decoupled input sequence only masks or suppresses target-irrelevant visual tokens during the action-side attention calculation process, so that the constraint mainly acts on the action generation path without directly destroying the underlying visual representation of the visual language action model.

[0051] For example, when the first Relevance score between visual tokens and target objects When it is lower, it indicates that the first If the visual token is not strongly correlated with the target object, then the corresponding token-level relevance score will be... It is also low. (Based on token-level relevance score) When the attention score of a visual token in the decoupled input sequence is modulated, the attention score of that visual token is lowered accordingly. When the action expert module generates an action block sequence based on the modulated decoupled input sequence, it suppresses or masks visual tokens with low attention scores to make more concentrated use of visual regions related to the target object, thereby reducing the interference of irrelevant visual tokens on the action generation process.

[0052] The action expert module is used to perform conditional diffusion generation processing on the modulated and decoupled input sequence to obtain a short-time action block sequence.

[0053] In one exemplary embodiment, the motion expert module is a Transformer-based diffusion motion generation model with the following loss function: ; ; in, Indicates loss of action, This represents random noise added to the action sequence. This represents the noise predicted by the motion expert module. Indicates the diffusion time step. This represents the condition context, which includes the state token and the anchor token. Uncertainty weights are used to balance the loss generated by the action; Indicates the weighting coefficient; Expressing expectations; This represents the spread time step in the short-time action block sequence output by the motion expert module. The corresponding single-step action.

[0054] In one exemplary embodiment, the motion expert module is specifically used for: (1) Actively query the anchor tokens in the modulated and decoupled input sequence to obtain target spatial positioning information.

[0055] (2) Based on the target spatial positioning information, perform each diffusion time step The noise component in the current noisy action block is predicted, and represented as: ; in, This represents a noise prediction network, specifically a Transformer-based diffusion-based action generation model. This represents the predicted value from the noise prediction network; Indicates diffusion time step The corresponding action block; This refers to the modulated and decoupled input sequence.

[0056] (3) For the predicted value Perform stepwise denoising to obtain the previous diffusion time step. The corresponding action block is represented as: ; in, Indicates the previous diffusion time step The corresponding action block, This represents a single-step denoising function. Indicates diffusion time step The corresponding action block.

[0057] (4) After passing through from arrive After multi-step iterative denoising, the final executable short-time action block sequence is obtained, represented as: ; in, This indicates the action block corresponding to diffusion time step 0. Indicates time A single-step action within an action block.

[0058] In the above process, anchor token It always resides between the state token and the action token. Since the action expert module can directly query this anchor token through the attention mechanism in each denoising step, the target space localization information can continuously constrain the action generation process. Due to the special nature of the sequence structure, when the model needs to generate or update... When any token in the set is selected, its attention calculation process will be forced to involve and traverse the tokens located in front of it. Tokens. This is because in Transformer multi-head attention, each output token needs to have its attention weights on all input tokens in the sequence calculated and then summed in a weighted manner. As part of the sequence, its rich spatial features are naturally integrated. This structured information bottleneck makes the generation path that ignores the target location more costly, thus prompting the policy network to faithfully utilize the perception grounding results provided by the perception grounding expert module to achieve stable transmission from where to look to how to act, and improve the accuracy of target selection and action execution in multi-object interference scenarios.

[0059] Compared to existing technologies where visual signals may be ignored, this application, through the constructed information bottleneck, creates an unavoidable causal dependency for the action expert during the reduction loss and denoising learning process: ignoring... This will directly lead to a sharp decline in the quality of the generated action features, resulting in inaccurate noise predictions and ultimately failing to reconstruct valid action trajectories. Therefore, the model is forced to actively query and pay close attention to the anchor token to extract the target spatial localization information it contains. At the same time, the state token at the beginning of the sequence also provides necessary ontology awareness. The two combined serve as conditional constraints for the denoising process.

[0060] The aforementioned decoupled anchor guidance mechanism and modulation structure not only serve the current architecture but can also be seamlessly integrated into other robot control models based on Transformer or Diffusion Policy as a portable, general-purpose module. This characteristic enables the technology to be widely applied in a broader range of embodied intelligence systems to generate more goal-oriented and trajectory-accurate motion sequences.

[0061] The action sequence generation method for multiple objects described in this application includes: S1: Acquire visual observation images of the robot operation scene, natural language text commands, and the body state information of the robotic arm; S2: Obtain a fused visual feature map based on the visual observation image and the natural language text instruction, and obtain a visual token based on the fused visual feature map; S3: Obtain the normalized bounding box coordinates of the target object in the visual observation image based on the fused visual feature map; S4: Encode and project the normalized bounding box coordinates to obtain anchor tokens, and construct a decoupled input sequence according to the order of the visual token, state token, anchor token, and action token; wherein, the state token is obtained according to the robot's current state; during the training phase, the action token is composed of real action blocks with added noise, and during the inference phase, the action token is composed of random noise; S5: Modulate the attention score of the visual token in the decoupled input sequence to obtain the modulated decoupled input sequence; S6: Perform conditional diffusion generation processing on the modulated and decoupled input sequence to obtain a short-time action block sequence.

[0062] The multi-object robot control method described in this application converts the final short-time action block sequence into low-level robot control instructions through an execution layer to control the robot's operation.

[0063] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. A multi-object action sequence generation system, characterized in that, include: The data acquisition module acquires visual observation images of the robot's operation scene, natural language text commands, and the robot arm's body state information. A visual language perception backbone network obtains a fused visual feature map based on the visual observation image and the natural language text instruction, and obtains a visual token based on the fused visual feature map; The sensing grounding expert module obtains the normalized bounding box coordinates of the target object in the visual observation image based on the fused visual feature map; The decoupling guidance module encodes and projects the normalized bounding box coordinates to obtain anchor tokens, and constructs a decoupling input sequence based on the order of the visual token, state token, anchor token, and action token; wherein, the state token is obtained through the robot's body state information; during the training phase, the action token is composed of real action blocks with added noise, and during the inference phase, the action token is composed of random noise; The modulation module modulates the attention score of the visual token in the decoupled input sequence to obtain the modulated decoupled input sequence; The action expert module performs conditional diffusion generation processing on the modulated and decoupled input sequence to obtain a short-time action block sequence; The sensing grounding expert module includes: The convolutional head performs structured dimensionality reduction processing on the fused visual feature map to obtain a dimensionality-reduced feature vector; wherein, the convolutional head includes a 2D convolutional layer, a batch normalization layer, a ReLU activation function and a flattening layer connected in sequence; A multilayer perceptron performs coordinate regression calculation on the dimensionality-reduced feature vector to obtain the bounding box coordinates of the target object in the visual observation image, and normalizes the bounding box coordinates to obtain normalized bounding box coordinates. The decoupling guidance module includes: The encoder maps the low-dimensional normalized bounding box coordinates to a high-dimensional embedding vector in the high-dimensional latent space. The projector projects the high-dimensional embedding vector onto the lexical dimension in the action expert module to obtain anchor tokens. The sequence construction unit constructs a decoupled input sequence based on the order of the visual token, the state token, the anchor token, and the action token, as follows: ;in, Indicates a visual token. Indicates a status token. Indicates anchor token, Indicates an action token; The modulation module is specifically used to: modulate the attention score of the visual token in the decoupled input sequence to obtain a modulated decoupled input sequence; For the The relevance scores between each visual token and the target object are calculated to obtain a set of visual token relevance scores, represented as follows: ， ， ； ； in, Indicates the first A relevance score between a visual token and the target object. Indicates the first A visual token This indicates that the visual token and the anchor token are concatenated using features. This represents a multilayer perceptron used to calculate the correlation score. This indicates that normalization is performed using the Sigmoid function; Represents the set of visual token relevance scores; The token-level relevance score is calculated based on the visual token relevance score set and is expressed as follows: ； in, Indicates from action token To visual token Token-level relevance score, The query vector representing the action token. Represent the key vector and value vector of a visual token; Indicates the shielding strength. express and The dimension; The attention score of the visual token in the decoupled input sequence is modulated based on the token-level relevance score to obtain the modulated decoupled input sequence.

2. The action sequence generation system as described in claim 1, characterized in that, The visual language perception backbone network includes: The trained visual language action model performs cross-modal feature fusion processing on the visual observation image and the natural language text instruction to obtain a fused visual feature map that is semantically aligned with the natural language text instruction.

3. The action sequence generation system as described in claim 2, characterized in that, The loss function of the sensing grounding expert module during the training process is expressed as: ； ， ； in, This represents the explicit target localization loss function of the sensing grounding expert module. Indicates GIoU loss, Indicates the coordinates of the predicted target bounding box. Represents the coordinates of the actual target bounding box; This represents the learnable parameters of the sensing grounding expert module; This represents the fused visual feature map; This represents the bounding box regression function implemented by the sensing grounding expert module. It represents the four coordinate components of the target bounding box coordinates in a four-dimensional real coordinate space.

4. The action sequence generation system as described in claim 3, characterized in that, The motion expert module is a Transformer-based diffusion motion generation model, and its loss function is: ； ； in, Indicates loss of action, This represents random noise added to the action sequence. This represents the noise predicted by the motion expert module. Indicates the diffusion time step. This represents the condition context, which includes the state token and the anchor token. Uncertainty weights are used to balance the loss generated by the action; Indicates the weighting coefficient; Expressing expectations; This represents the spread time step in the short-time action block sequence output by the motion expert module. The corresponding single-step action.

5. The action sequence generation system as described in claim 4, characterized in that, The motion expert module is specifically used for: Actively query the anchor tokens in the modulated and decoupled input sequence to obtain target spatial positioning information; Based on the target spatial positioning information, for each diffusion time step The noise component in the current noisy action block is predicted, and represented as: ； in, This represents a noise prediction network, specifically a Transformer-based diffusion-based action generation model. This represents the predicted value from the noise prediction network; Indicates diffusion time step The corresponding action block; This represents the modulated and decoupled input sequence; For predicted values Perform stepwise denoising to obtain the previous diffusion time step. The corresponding action block is represented as: ； in, Indicates the previous diffusion time step The corresponding action block, This represents a single-step denoising function. Indicates diffusion time step The corresponding action block; After from arrive After multi-step iterative denoising, the final executable short-time action block sequence is obtained, represented as: ； in, This indicates the action block corresponding to diffusion time step 0. Indicates time A single-step action within an action block.

6. A method for generating action sequences for multiple objects, wherein the method is implemented using the action sequence generation system described in any one of claims 1-5, characterized in that, The action sequence generation method includes: The system acquires visual observation images, natural language text commands, and the body state information of the robotic arm in the robot's operation scene. A fused visual feature map is obtained based on the visual observation image and the natural language text instruction, and a visual token is obtained based on the fused visual feature map; The normalized bounding box coordinates of the target object in the visual observation image are obtained based on the fused visual feature map. The normalized bounding box coordinates are encoded and projected to obtain anchor tokens. A decoupled input sequence is constructed according to the order of the visual token, state token, anchor token, and action token. The state token is obtained based on the robot's current state. During the training phase, the action token is composed of real action blocks with added noise, and during the inference phase, the action token is composed of random noise. The attention score of the visual token in the decoupled input sequence is modulated to obtain the modulated decoupled input sequence; The modulated and decoupled input sequence is subjected to conditional diffusion generation to obtain a short-time action block sequence.

7. A robot control method for multiple objects, characterized in that, The robot's operation is controlled by the short-time action block sequence generated by the action sequence generation system according to any one of claims 1-5.

Images