A Long-Range Robot Manipulation Method Based on Instruction-Conditioned Feature Learning

By using a method based on instruction-conditional feature learning, and utilizing multi-view visual data and operation instructions, a position query sequence is constructed, which solves the problem of interference from multiple visual objects in long-range robot operations and achieves efficient and accurate operation in complex scenarios.

CN123103667APending Publication Date: 2026-11-17SUZHOU TIANYUANXING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611498790.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing long-range robot operation methods struggle to effectively handle multiple semantically valid and visually operable foreground objects when faced with scene changes caused by the execution of preceding skills, operations by external collaborators, or dynamic environmental changes, leading to operation failures.

Method used

By using a method based on instruction-conditional feature learning, multi-view visual data and operation instructions are acquired, a location query sequence is constructed, and a multimodal attention network is used to extract instruction-conditional visual information. The location query representation is then trained through contrastive learning and consistency constraints to generate a clear action sequence, avoiding direct access to the original visual features.

Benefits of technology

This method can improve the robustness of long-range robot operations without increasing computational overhead and training costs, ensuring that the generated actions are consistent with the current operation instructions and adapting to complex scene changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103667A_ABST
    Figure CN123103667A_ABST
Patent Text Reader

Abstract

This disclosure relates to the field of robot control technology, specifically to a long-range robot operation method based on instruction-conditional feature learning. Addressing the problem of observation space shift caused by the execution of preceding skills in long-range operations, this disclosure constructs a bottleneck pathway between the feature extraction backbone of the vision-language-action model and the action generation module, using a position query sequence to carry visual evidence. Attention masks are used to restrict the action generation module to obtain visual information that satisfies instruction conditions only through this pathway. During the training phase, instruction comparison and offset consistency constraints are used to jointly optimize the position query representation, making it sensitive to instructions and stable to scene offset changes. During the inference phase, counterfactual operation instructions are constructed for interfering objects in the scene, and the output action is corrected based on the difference between these instructions and the target instruction action sequence. Training and inference form a closed loop, requiring no additional trainable parameters, providing targeted robustness for the execution of long-range skill combinations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of robot control technology, and more specifically to a long-range robot operation method based on instruction-conditional feature learning. Background Technology

[0002] In robotic operation tasks, training a vision-language-action model through imitation learning on short-term, single-step skill demonstrations, and then connecting several skills sequentially to form a long-term operational skill chain through task planning, is a common approach in the current industrial and service robotics field for achieving complex tasks. In this approach, each skill is typically trained independently based only on its corresponding demonstration data, and the changes caused by preceding skills in the skill chain to the scene are not explicitly modeled during training and deployment. Existing technologies for improving the robustness of such strategies mainly employ two approaches: one is to enhance the invariance of the strategy to low-level visual perturbations by adding data augmentation techniques such as image noise, lighting changes, and viewpoint perturbations to the training data; the other is to use generative action heads such as flow matching and diffusion models to improve the action prediction accuracy and multimodal coverage of single-step skills, and to compress the scale of visual features input to the action head through spatial attention pruning and region feature filtering. These existing technologies have the following shortcomings: First, data augmentation methods model perturbations limited to low-level visual noise, failing to cover semantically valid and visually still operable foreground object changes that remain or are introduced into the scene due to preceding skill execution, external collaborator operations, or dynamic environmental changes. Second, while feature pruning methods can compress the scale of visual features, pruning is usually based on spatial saliency or fixed region priors, without establishing an explicit association with the current operation instruction. This makes it difficult to selectively retain evidence related to the instruction when multiple candidate operation objects exist simultaneously in the same scene. Third, existing strategies generally employ end-to-end training, with action prediction directly coupled to the complete visual observation. When multiple semantically valid foreground objects remain in the scene at the later stages of the skill chain, confusion can easily arise between candidate objects, leading to operation failure. Summary of the Invention

[0003] This disclosure provides a long-range robot operation method based on instruction-conditional feature learning, which aims to solve the problems mentioned in the background art.

[0004] The technical solution adopted in this disclosure is as follows: A long-range robot operation method based on instruction-conditional feature learning includes: The robot acquires multi-view visual data and corresponding operation instructions collected during the execution of skill sequences, performs visual encoding on the multi-view visual data to obtain visual feature sequences for each view, and performs language encoding on the operation instructions to obtain instruction features. A location query sequence corresponding to each view is constructed, the location query sequence including a learnable query vector and its location information encoding; the location query sequence, visual feature sequence and instruction features are input into a multimodal attention network, and the action generation module is limited by an attention mask to obtain instruction-conditional visual information only through the location query sequence, and direct access to the original visual feature sequence is prohibited; Based on the operation instructions and their variants in the same scene, multi-view visual data, and offset visual data caused by prior skills, the position query representation is trained through contrastive learning and consistency constraints; and the above training is jointly optimized with the action generation loss based on multi-view visual data and expert demonstration action trajectories to optimize each network parameter. Visual data of the interfering object introduced by the preceding skills and the target operation command are acquired. Based on the semantic information corresponding to the interfering object in the visual data, a counterfactual operation command pointing to the interfering object is constructed. The visual data is combined with the target operation command and the counterfactual operation command respectively and input into the multimodal attention network to obtain the corresponding position query representations. Each position query representation and the corresponding command are input into the action generation module. Through the multimodal attention network and the action generation module, in the process of generating the flow matching numerical integration, the target velocity field is guided by extrapolation and integral update based on the difference vector between the predicted velocity fields corresponding to the target operation command and the counterfactual operation command respectively, and the final action sequence for controlling the robot actuator is output.

[0005] Preferably, the step of performing visual encoding and language encoding to obtain visual feature sequences and instruction features for each view includes: dividing the image blocks according to the image frames corresponding to each view; mapping each image block to the visual embedding space via linear projection; superimposing two-dimensional position encoding on the corresponding visual embedding based on the two-dimensional position of the image block in the corresponding image frame; performing hierarchical feature interaction on the embedding of each image block within the same view through the multi-head self-attention mechanism of the visual Transformer to obtain a view visual feature sequence containing spatial position information of the image blocks; and superimposing view identifier encoding for different views to distinguish the source of the view; performing sub-word segmentation on the operation instruction and mapping it to a word embedding sequence; superimposing word order position encoding on the word embedding sequence; establishing contextual dependencies between different instruction terms through the multi-head self-attention mechanism of the language Transformer; and mapping the output via linear projection to a multimodal embedding space that matches the visual feature sequence.

[0006] Preferably, the construction of the location query sequence corresponding to each view includes: For each view, a set of independent learnable query vectors is set, and the number of learnable query vectors is less than the length of the visual feature sequence of the corresponding view; Each learnable query vector is superimposed with the view position code representing the view to which it belongs and the position code representing the query slot, respectively, to obtain the position query sequence; The query sequence of each view location is concatenated with the visual feature sequence and the instruction feature sequence to form a joint multimodal sequence, which is then input into a multimodal attention network for bidirectional self-attention calculation. This allows each query vector to extract key features of the operation target from the visual features based on the instruction semantic information. The visual features are then weighted and aggregated according to the attention weights between each query vector and the visual features to obtain a location query representation that carries visual condition information.

[0007] Preferably, the action generation module that limits the action generation by attention masking obtains visual information only through the location query sequence, including: The attention interaction relationships between the visual feature sequence, the location query sequence, and the instruction feature are configured using an attention mask matrix. The query end corresponding to the location query sequence is configured to allow access to the visual feature sequence and the instruction feature. The action generation module is independent of the multimodal attention network and takes the location query representation and instruction feature output by the multimodal attention network after processing the location query sequence as input, without directly accessing the original visual feature sequence.

[0008] Preferably, based on the operation instructions and their variants in the same scenario, a location query representation is trained through comparative learning, including: The instruction variants include a rewrite instruction that points to the same operation object and the same target location as the operation instruction, and a counterfactual operation instruction that points to a different operation object than the operation instruction; the operation instruction, rewrite instruction, and counterfactual operation instruction are respectively input into a multimodal attention network to obtain corresponding location query representations; The position query representations corresponding to the operation instruction and the rewrite instruction are globally pooled, mapped by a projection network, and then the positive sample similarity constraint between them is constructed using InfoNCE contrast loss. The global position query representation corresponding to the counterfactual operation instruction is mapped by the projection network and used as a negative sample in the calculation of InfoNCE contrast loss. For each query slot in the location query sequence, the query representation corresponding to the query slot under the operation instruction, the rewrite instruction and the counterfactual operation instruction is extracted respectively. After being mapped by the projection network, the slot-level InfoNCE contrast loss is used to constrain the similarity of the representation of the same query slot under the operation instruction and the rewrite instruction, as well as the similarity interval with the corresponding representation under the counterfactual operation instruction. The instruction-conditional feature learning loss is obtained by weighted summing of the global InfoNCE contrast loss and the slot-level InfoNCE contrast loss of each query slot.

[0009] Preferably, the method for acquiring the offset visual data caused by the preceding skill is as follows: According to the skill sequence corresponding to the robot's long-range operation, at least one preceding skill is executed before the current target operation, so that the changes in the spatial state of objects in the scene generated during the execution of the preceding skill are retained in the current operation scene. After the preceding skill is completed and the robotic arm returns to the preset return configuration, offset visual data is collected through a visual sensor corresponding to the multi-view visual data. Based on the multi-view visual data and the offset visual data, a position query representation is trained through consistency constraints, including: The multi-view visual data and offset visual data are respectively input into a visual encoding network and a multimodal attention network with shared parameters to obtain a first position query representation and a second position query representation with the same query slot correspondence, and are respectively mapped to the consistency discrimination space through a projection head with shared parameters. The projection result corresponding to the first position query representation is used as the stopping gradient target branch, and the projection result corresponding to the second position query representation is used as the gradient update branch. Cosine similarity loss is used to constrain the two projection representations corresponding to the same view and the same query slot, and the two projection representations are aggregated according to the cosine similarity loss of each query slot to obtain the offset consistency loss.

[0010] Preferably, the action generation module employs a flow-matching action generation mechanism. It represents the robot expert demonstration action trajectory as a continuous action trajectory, constructs a continuous-time interpolation state starting from a random noise action trajectory with the same dimension as the expert demonstration action trajectory, and constructs a target velocity field corresponding to the time state based on the vector difference between the random noise action trajectory and the expert demonstration action trajectory. The position query representation, the instruction features, the continuous-time interpolation state, and the corresponding time code are input into the action generation network. The action generation network predicts the velocity field corresponding to the continuous-time interpolation state, and the flow-matching action generation loss is established using the squared error between the predicted velocity field and the target velocity field. By jointly optimizing the velocity fields under different continuous-time states, a conditional action trajectory generation model is obtained.

[0011] Preferably, in the process of generating the numerical integral of the flow matching, the target velocity field is guided by extrapolation and integral update based on the difference vector between the predicted velocity fields corresponding to the target operation command and the counterfactual operation command, respectively, to output the final action sequence for controlling the robot actuator, including: According to the corresponding integration time step and the corresponding action dimension, the action velocity fields corresponding to the target operation command and the counterfactual operation command are paired respectively, and the difference vector between the target action velocity field vector and the counterfactual action velocity field vector corresponding to each integration time step is calculated. The difference vector of each integration time step forms the action velocity field difference sequence. The difference vectors of each integral time step in the motion velocity field difference sequence are weighted according to the steering weight to obtain the velocity field correction vector of the corresponding integral time step. The velocity field correction vectors of each integral time step form a velocity field correction sequence with the same integral time step and motion dimension as the target motion velocity field. The target motion velocity field and the velocity field correction sequence are linearly combined according to the corresponding integration time step to obtain the guided velocity field. The guided velocity field is then numerically integrated according to the preset integration step size to output the final motion sequence.

[0012] Preferably, when the acquired visual data contains multiple candidate interference objects, a counterfactual operation instruction pointing to each candidate interference object is constructed. Each counterfactual operation instruction is then combined with the visual data and input into the multimodal attention network and the action generation module to obtain candidate counterfactual action velocity fields corresponding to each candidate interference object. Candidate difference vectors between the target action velocity field and each candidate counterfactual action velocity field are calculated according to the corresponding integral time step and corresponding action dimension. Based on the attention weight between the query slot representation corresponding to each candidate interference object in the multimodal attention network and the instruction feature corresponding to the target operation instruction, the relevance weight of each candidate interference object relative to the current operation scene is determined. Weighted aggregation is performed on each candidate difference vector using the relevance weight to obtain the action velocity field difference sequence.

[0013] Compared with the prior art, the beneficial effects of this disclosure include the following aspects: First, the training supervision signals relied upon by this method, namely the instruction variant comparison signal and the scene offset consistency signal, can be constructed by themselves based on the existing demonstration data of single-step short-term skills and their scene variants. It does not require additional collection or annotation of long-term operation demonstrations covering the complete skill chain. It can directly reuse the demonstration data resources accumulated under the existing short-term skill training system, avoiding the data scale bottleneck caused by the high manual cost of long-term demonstration data collection and the large number of combinable skills.

[0014] Second, the motion correction process reuses the same set of pre-trained multimodal attention network and motion generation module for two conditional forward computations. In the inference stage, the visual encoding features of the multi-view visual data only need to be calculated once. The two conditional forward computations corresponding to the target operation command and the counterfactual operation command are shared. The conditional feature extraction and velocity field prediction are completed in parallel through batch processing, avoiding the computational redundancy caused by repeated calculation of visual features. No new trainable parameters are added, and no special fine-tuning or retraining is required for specific deployment scenarios. It can be embedded into the existing vision-language-motion model inference link that has been trained and deployed with low modification cost, adapting to the dual constraints of policy iteration cycle and computing power consumption in robot application scenarios.

[0015] Third, unlike the conventional approach of indirectly addressing observational disturbances by improving the overall noise resistance of features, this method directly targets the modeling object as scene remnants introduced during skill chain execution or external environment changes that are unrelated to the current instruction but still visually operable. By using location query bottlenecks and instruction conditional constraints, the visual evidence on which action prediction is based is converged to the range related to the current instruction. When multiple visually salient candidate operation objects appear consecutively at the end of the skill chain, this mechanism can still provide a source of visual evidence that is clearly directed and consistent with the current operation instruction for action generation. Attached image description: To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A schematic diagram of the overall process of the method provided in this disclosure is shown; Figure 2 This diagram illustrates the hardware architecture of the application environment provided in this disclosure; Figure 3 This illustration shows a schematic diagram of the location query bottleneck and attention mask structure provided in this disclosure; Figure 4 A schematic diagram of offset consistency learning during the training phase provided in this disclosure is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. In practice, the application environment on which the method described in this disclosure depends includes the robot body, a vision sensor group, a computing unit, a robot controller, and a communication link. For ease of description, the interference object referred to in this disclosure refers to an object in the scene that is unrelated to the current operation command but still visually operable. Its source is not limited to the residue of the previous skill execution, but can also be an object introduced or changed by external collaborators, other operating mechanisms, or dynamic environmental changes. It can also be called a scene residue object or a candidate operation object. The position query representation, also known as the position query output feature, refers to the output representation obtained after the position query sequence is processed by the multimodal attention network. The turning weight, also known as the correction intensity coefficient, is used to characterize the proportion of the counterfactual action difference sequence in the final action sequence correction process. The above synonyms have the same meaning in this specification and claims, and do not refer to different technical features due to different expressions. In addition, the action speed field and speed field are the same technical feature, but are different names for each other at different levels of expression.

[0017] The robot body can be a single-arm robotic arm or a dual-arm collaborative robot, each equipped with an end effector. The end effector can be one or more combinations of a parallel gripper, a vacuum suction cup, or a multi-fingered dexterous hand, specifically determined by the shape of the object to be manipulated and the grasping method. The vision sensor group includes at least two industrial cameras mounted at different locations on the robot body. These can be RGB cameras or RGB-D cameras that simultaneously output depth information. Mounting locations include, but are not limited to, the robot wrist, the top of the robot body, and the front side of the worktable, used to acquire images of the operating scene from different perspectives, forming the multi-view visual data described in this disclosure. The computing unit is equipped with a graphics processor to carry and run the program code corresponding to the visual encoding network, language encoding network, multimodal attention network, and motion generation module. Its deployment location can be an edge computing device installed near the robot body or worktable, a local server connected to the robot body via a local area network or industrial Ethernet, or a cloud server connected to the robot body via a wireless or wired wide area network. The robot controller communicates with the computing unit to receive the final motion sequence output by the computing unit, converts it into drive commands for each joint of the robot body and the end effector after kinematic calculation. The input methods for operation commands include speech recognition to text, text input via a teach pendant, or automatic issuance by the host task planning system, and are not limited to any specific input channel. Data transmission links are established between the vision sensor group and the computing unit, and between the computing unit and the robot controller, respectively, via industrial Ethernet, USB interface, or wireless LAN.

[0018] As an example, see Figure 2 As shown, the application environment on which the method described in this disclosure depends includes a robot body, a vision sensor group consisting of multiple cameras, a computing unit carrying each network module, a robot controller, and communication links between the various parts; the multi-view vision data collected by the vision sensor group and the operation instructions received by the operation instruction input terminal are transmitted to the computing unit together, and the final action sequence output by the computing unit is transmitted to the robot controller, which then calculates and drives the robot body to complete the corresponding operation.

[0019] In a specific deployment scenario based on a dual-arm collaborative robot, the robot body is a dual-arm collaborative robot, each robotic arm has six degrees of freedom, and each end is equipped with a parallel gripper. The vision sensor group includes three cameras, which can be RGB cameras or RGB-D cameras. One camera is fixedly mounted on the robot's head, looking down at the worktable. The other two are wrist-eye cameras mounted on the wrists of the two robotic arms, moving with the corresponding end effectors of the robotic arms. In another specific deployment scenario based on a single-arm robotic arm, the robot body is a single-arm robotic arm, and the vision sensor group is simplified to two cameras: a top-view camera fixedly mounted on the robot's head and a wrist-eye camera mounted on the wrist of the robotic arm, forming the dual-view described in this disclosure. The remaining processing flow is consistent with the aforementioned dual-arm collaborative robot scenario, thus forming the multi-view described in this disclosure. The computing unit is an edge computing device installed inside the robot body base, connected to the three cameras and the robot controller via a gigabit industrial Ethernet. Hereinafter, the "counterfactual operation instruction" of this disclosure will be abbreviated as counterfactual instruction, and the "target operation instruction" will be abbreviated as target instruction. Subsequent embodiments will use this abbreviation. It should be understood that the operation instructions mentioned in the training phase refer to the operation instructions in the training samples corresponding to the collected single-step skill demonstrations, which can be the instructions corresponding to any skill in the skill chain; the target operation instructions mentioned in the inference phase specifically refer to the operation instructions corresponding to the skill to be executed. There is no essential difference between the two in terms of language form and encoding method. Both are processed by the aforementioned language encoding network, and only the methodological stage and the specific skills they refer to are different.

[0020] During the training phase, several single-step demonstration trajectories were collected for preset desktop operation primitive skills, including grasping, placing, opening and closing container lids, and pouring materials. Each demonstration trajectory corresponds to a natural language operation instruction, a three-view image at the corresponding moment, and the pose trajectory of the robotic arm's end effector. For each scenario, the operator or a preset template rewrote the original instruction semantically and selected counterfactual instructions from other operable objects in the scenario. The three types of instructions, together with the three-view image of the scenario, constitute a set of instruction comparison samples.

[0021] Offset visual data is obtained through the actual execution of preceding primitive skills: before capturing a demonstration of a target skill, at least one other primitive skill is executed in the same workbench scene. After the preceding skill is completed and the robotic arm returns to the preset return configuration, the three cameras capture the three-view images at this time, which serve as the offset visual data corresponding to the target skill. These images, along with the three-view images captured before the execution of the preceding skill, form a pairing sample. The computing unit performs joint training of the visual encoding network, language encoding network, multimodal attention network, and action generation module based on the above samples. The network parameters after training are fixed and deployed on the edge computing device. Optionally, the paired offset visual data can also be generated through offline replay: based on the task environment description configuration file, the state of task-independent entities is modified or non-target entities are added or removed to construct a variant environment. The original demonstration action trajectory is then replayed in this variant environment, and the observation images during the execution process are recorded to construct offline paired samples, without the need for additional online execution and acquisition.

[0022] During the reasoning phase, the robotic arm executes several primitive skills in the order of the skill chain. Before each skill is executed, the three cameras capture three-view images of the current scene, and the operation instructions corresponding to the skill are input into the deployed network. If there is an object left over from the execution of the next preceding skill in the current scene, for example, if the previous skill is "pouring materials into a container", the material left in the container constitutes an interference object for the next skill "taking the tool next to the container", then the computing unit constructs a counterfactual instruction pointing to the interference object, obtains the corrected final action sequence, and drives the robotic arm to complete the corresponding operation after being calculated by the robot controller.

[0023] In another embodiment, the method described herein is applied to a production line incoming material sorting scenario, where the long-range operation task is composed of three primitive skills in sequence: The process involves three steps: first, grabbing incoming workpieces from the upstream conveyor belt and placing them into a temporary storage box; second, removing qualified workpieces from the temporary storage box and placing them into a downstream turnover box; and third, using a labeling tool placed on one side of the workbench to label the workpieces in the turnover box.

[0024] After the first skill is completed, a workpiece remains in the temporary storage box. For the second skill, this workpiece is the object to be operated and does not constitute interference. However, for the third skill, the temporary storage box and its internal workpieces, as well as the workpieces added to the turnover box after the second skill is executed, are all residual objects in the scene that are unrelated to the operation instruction of "taking the labeling tool" but are still visually operable, constituting the interference objects described in this disclosure. When the computing unit executes the inference corresponding to the third skill, it constructs a counterfactual instruction, such as "taking the workpiece in the turnover box," based on the aforementioned interference objects present in the current three-view image. This counterfactual instruction, along with the target instruction "taking the labeling tool," is input into the trained multimodal attention network and the action generation module, respectively, to obtain two sets of action predictions. The action sequence corresponding to the target instruction is then corrected based on the difference between the two predictions.

[0025] As an example, in some more specific scenarios, the training and inference of the method described in this disclosure can be deployed on different computing units: the sample construction and network joint optimization involved in the training phase are completed on a server cluster with multiple graphics processors, and the network parameters after training are quantized and compressed and then sent to the edge computing device deployed on the robot body; the inference phase only runs forward computation on the edge computing device and no longer depends on the server cluster.

[0026] In this deployment method, the same set of trained network parameters can be distributed to different robot bodies with the same end effector interface, including single-arm robotic arms and dual-arm collaborative robots. For single-arm robotic arms, their vision sensor group is simplified to one or two cameras, and the number of views corresponding to the position query sequence in the multimodal attention network is adjusted accordingly, while the rest of the processing flow remains the same.

[0027] It should be noted that in actual deployment scenarios, multiple candidate interference objects may exist simultaneously in the same operation scenario. Taking the aforementioned production line sorting scenario as an example, if before the execution of the third skill, there is another detection fixture that is not involved in this skill chain temporarily placed on the workbench, the computing unit constructs counterfactual instructions for the workpiece in the temporary storage box, the workpiece in the turnover box, and the detection fixture, respectively, and obtains three sets of counterfactual action sequences. The computing unit determines the relevance weight of each candidate interference object relative to the current scenario based on the attention weight between the query slot representation corresponding to each candidate interference object in the multimodal attention network and the target instruction feature, and uses this weight to perform weighted aggregation of the three sets of counterfactual action difference vectors to obtain the counterfactual action difference sequence used to correct the target action sequence.

[0028] Relatedly, the method for determining the turning weight is as follows: Under the same target instruction and the same scene input conditions, the computing unit integrates from multiple independently sampled random noise action trajectories based on the flow matching action generation mechanism to obtain multiple candidate target action trajectories. The prediction uncertainty measure is obtained by calculating the degree of dispersion of the distribution of these multiple candidate trajectories at each time step and each action dimension, and the turning weight value under the current scene is determined accordingly.

[0029] To ensure full disclosure of the technical solution of this disclosure, the following description, in conjunction with the accompanying drawings, further illustrates the long-range robot operation method based on instruction-conditional feature learning provided in this disclosure; the specific configurations of the robot body, vision sensor group, computing unit, and robot controller in the aforementioned embodiments will not be repeated. The following description, in conjunction with the aforementioned dual-arm collaborative robot and production line sorting scenario, further explains the specific implementation of each step of the method described in this disclosure.

[0030] As an example, see Figure 1 As shown, the method described in this disclosure sequentially undergoes instruction comparison learning, offset consistency learning, action generation learning, and joint optimization of network parameters during the training phase; and sequentially undergoes interference object identification, counterfactual operation instruction construction, target action sequence and counterfactual action sequence generation, and action difference correction during the inference phase, finally outputting the final action sequence used to control the robot actuator. The training and inference phases share the same pre-constructed position query sequence and multimodal attention network. The following parts are arranged according to... Figure 1 The process flow shown is explained in detail below.

[0031] Visual encoding can be implemented by a visual encoding network, which can adopt a visual Transformer architecture or a convolutional neural network architecture; language encoding can be implemented by a language encoding network, which can adopt a pre-trained language model or other network structures with sequence modeling capabilities, and this disclosure does not limit this; at the same time, the semantic information corresponding to the interference object mentioned in this disclosure, that is, the object category information corresponding to the interference object in the visual data, can be obtained through conventional object detection networks or other methods, and is not limited.

[0032] The visual feature sequences and instruction features obtained after processing by the two networks are mapped to the same multimodal feature space in terms of feature dimension, so that they can be uniformly input into the multimodal attention network in the future. The length of the visual feature sequence is related to the image resolution and the downsampling ratio of the visual coding network, and the length of the instruction features is related to the number of words segmented in the operation instruction. Both vary with the specific input and are not subject to fixed limitations.

[0033] The location query sequence consists of a set of learnable query vectors superimposed with corresponding location information encoding. This set of learnable query vectors can be randomly initialized before training begins, for example, by initializing them according to a normal distribution with a mean of zero and a preset variance, and then gradually updated with backpropagation.

[0034] For each input view, the number of learnable query vectors is a pre-set hyperparameter, which can be set to sixteen to thirty-two for example. In practice, the relationship between this number and the length of the visual feature sequence can be determined through ablation experiments based on the specific backbone network and the number of candidate operation objects in the operation scenario.

[0035] The number of views included in the visual sensor group is not limited to the three in the aforementioned embodiments; it can be two or more, depending on the number of cameras that can be installed on the robot body and the observation range of the operating scene. Each view corresponds to a set of independent position query vectors.

[0036] After the location query sequence, visual feature sequence, and command feature are input into the multimodal attention network, information interaction between them is achieved through the multi-head self-attention mechanism in the network. The action generation module, also known as the action decoder or action prediction head, uses the output representation corresponding to the location query sequence as its means of acquiring visually relevant information. Regarding the specific implementation of this information access relationship, taking the aforementioned dual-arm collaborative robot as an example, if at a certain moment the length of the visual feature sequence corresponding to each of the three views is P, the length of the command feature sequence is L, and the length of the location query sequence is Q, then the length of the complete sequence input into the multimodal attention network is 3P+L+Q. Before the multi-head self-attention calculation, the visibility between each query-key pair is configured by an attention mask matrix, the row and column dimensions of which are consistent with the length of the complete sequence.

[0037] For rows in the query terminal that correspond to the location query sequence, the values ​​in the corresponding columns of the visual feature sequence and instruction feature are configured to allow access; for rows in the query terminal that correspond to the action generation module, the values ​​in the corresponding columns of the original visual feature sequence are configured to prohibit access, and the values ​​in the corresponding columns of the location query sequence and instruction feature are configured to allow access.

[0038] In implementation, "access denied" can be achieved by setting the attention score at the corresponding position in the mask matrix to a minimum value before normalization, so that the attention weight corresponding to that part approaches zero after normalization. This completes the information path configuration where visual information is indirectly transmitted through the position query sequence, rather than the action generation module directly obtaining the original visual features. It should be understood that the connection relationship between the action generation module and the multimodal attention network is not limited to the above-mentioned query token participating in the same self-attention calculation; in another optional implementation, the action generation module can also be set as a downstream network independent of the multimodal attention network, only receiving the output representation corresponding to the position query sequence as its sole means of obtaining visually related information, which also satisfies the limitation that the action generation module only obtains visual information through the position query sequence.

[0039] Instruction variants include semantically equivalent rewritten instructions and counterfactual operation instructions. Taking the operation instruction "Place the workpiece in the temporary storage box into the turnover box" in the aforementioned production line sorting scenario as an example, its semantically equivalent rewritten instruction could be "Place the workpiece in the temporary storage box into the turnover box." Both refer to the same operation object and target location, differing only in their expression. Counterfactual operation instructions, on the other hand, refer to another operation object within the same scenario, such as "Take the labeling tool from one side of the workbench." The construction methods for these instruction variants include: pre-writing candidate instructions manually for the same scenario; automatically generating instructions by substituting the names of detected objects in the scenario into template slots based on a preset instruction template; or generating instructions using a language model, with the operation instruction and the current scenario description as input.

[0040] Contrastive learning and consistency constraints are used to train the location query representation, specifically: In contrastive learning, operation instructions, rewrite instructions, and counterfactual operation instructions in the same scenario are respectively input into the multimodal attention network to obtain their respective location query representations. The similarity between the corresponding representations of operation instructions and rewrite instructions is constrained to be higher than the similarity between the corresponding representations of operation instructions and counterfactual operation instructions by a contrastive loss function. This contrastive loss function can adopt the InfoNCE loss form commonly used in contrastive learning.

[0041] Regarding consistency constraints, multi-view visual data and offset visual data under the same operation command are respectively input into the multimodal attention network to obtain their respective position query representations. The similarity between the two is constrained by a consistency loss function, which can be a cosine similarity loss function. Optionally, the gradient backpropagation of one of the representations can be blocked. This processing method usually helps to avoid representation collapse and improve training stability in self-supervised learning.

[0042] The aforementioned contrast loss and consistency loss can be weighted and summed with the action generation loss according to preset weights to obtain the overall training loss. Gradient-based optimization algorithms such as Adam or AdamW can be used to update the network parameters of the visual encoding network, language encoding network, multimodal attention network, and action generation module. Optionally, the joint training can be performed in two stages: the first stage uses only the action generation loss for training, allowing randomly initialized position queries to gradually establish a stable attention pattern for the scene; the second stage introduces instruction contrast loss and offset consistency loss for joint optimization based on the first stage, and the weight coefficients of the two newly introduced losses can be gradually increased to preset values ​​using a linear warm-up method, followed by continued training until network convergence. For example, the first stage can be the first 10,000 steps of training, and the second stage can be the next 20,000 steps of training, with the linear warm-up steps for the weight coefficients set to 1,000 steps; for example, the initial learning rate can be set to 1×10⁻⁶. -4 The loss can be gradually decayed as the training rounds progress using methods such as cosine annealing. The batch size of the training samples used for a single parameter update can be determined based on the memory capacity of the computing unit. For example, it can be set to 16 or 32 sets of scene samples within the same batch. In this case, the negative samples within the batch in the aforementioned instruction comparison learning are taken from the corresponding representations of other scenes in the same batch, excluding the current scene. The weight coefficients of each loss term can be determined through hyperparameter search based on the training effect of the specific task. This disclosure does not limit this.

[0043] During the inference phase, the methods for determining interference objects, also known as scene residual objects, include: detecting foreground objects in the current visual data using a pre-set target detection network, and identifying objects in the detection results that are different from the operation objects corresponding to the target operation command as candidate interference objects; identifying operation objects involved in the preceding skills immediately adjacent to the current skill in the skill sequence execution record as interference objects; pre-specifying interference objects by the operator or the higher-level task planning system; or comparing the results of two adjacent visual data acquisitions and identifying objects that have undergone position or state changes but are unrelated to the current target operation command as candidate interference objects, including objects introduced by external collaborators or dynamic environmental changes. If no interference objects are identified in the current scene, alternatively, counterfactual operation commands may not be constructed, i.e., the weights of the difference vector are set to zero in the subsequent correction process, in which case the final action sequence is consistent with the target action sequence.

[0044] After identifying the interference target, a counterfactual operation instruction pointing to the interference target is constructed. This counterfactual operation instruction, along with the target operation instruction, is input into the multimodal attention network and the action generation module, respectively, to obtain a counterfactual action sequence and a target action sequence. These are represented as action vector sequences according to their corresponding time steps and dimensions. For example, the motion velocity field of the robotic arm's end effector can be represented by a seven-dimensional vector consisting of the velocity components corresponding to the six-degree-of-freedom pose increments and the gripper opening and closing speeds (for a dual-arm collaborative robot, the dimension of the motion velocity field vector can be extended to fourteen to thirty-two dimensions to respectively characterize the velocity components corresponding to the multi-degree-of-freedom joint states of each robotic arm and the gripper control quantities). The target motion velocity field vector at a certain integral time step is ( The counterfactual action velocity field vector is ( The difference vector for the integral time step is obtained by subtracting the corresponding dimensions of the two vectors. This difference vector is then multiplied by the steering weight to obtain the velocity field correction vector. The velocity field correction vector is then added to the corresponding dimension of the target motion velocity field vector to obtain the velocity field guided by the integral time step. The velocity fields guided by each integral time step form a guided velocity field sequence. This sequence is numerically integrated according to a preset integration step size to obtain the corrected final motion sequence. This final motion sequence is output to the robot controller, where kinematic calculations convert it into drive commands for each joint of the robot body and the end effector. The corresponding hardware configuration and control chain have been described above and will not be repeated here.

[0045] Regarding the specific application of the final action sequence at the robot controller execution level, a rolling time-domain execution method can be adopted: the robot controller drives the robot body to complete the corresponding actions only based on the action vectors corresponding to the first few time steps in the final action sequence. After the actions corresponding to these time steps are completed, the vision sensor group re-collects multi-view visual data of the current scene, and based on the re-collected visual data and the same target operation instruction, the aforementioned process of determining the interference object, constructing the counterfactual operation instruction, and correcting the action sequence is re-executed to obtain a new final action sequence. This cycle continues until the operation corresponding to the currently to be executed skill is completed. In another optional implementation, the actions corresponding to all time steps in the final action sequence can also be executed at once without re-observation and re-prediction during the execution process. The specific method can be determined based on the execution accuracy requirements of the operation object and the control cycle of the robot body. After a skill in the skill sequence is completed, the complete process of acquiring visual data, constructing the position query sequence, determining the interference object, and correcting the action sequence is repeated based on the operation instruction corresponding to the next skill in the skill sequence until all skills in the skill sequence are executed in sequence.

[0046] Furthermore, in conjunction with the aforementioned dual-arm collaborative robot embodiment, the specific implementation methods of visual encoding, language encoding, and location query sequence will be further explained.

[0047] In terms of visual coding, the visual coding network first divides the image frames corresponding to each view according to a preset image block size. For example, for an image frame with a resolution of 224×224 pixels, dividing it into image blocks of 16×16 pixels will result in 196 image blocks of 14×14 pixels. Optionally, for cases where the resolution and the image block size are not divisible, the image frame can be scaled or edge-padded to make the processed resolution satisfy the divisibility relationship.

[0048] Each image patch, after being flattened, is mapped to a visual embedding space of a preset dimension through a linear projection layer. For example, this dimension can be set to 768 dimensions. Two-dimensional positional encoding is generated based on the row and column coordinates of the image patch in the corresponding image frame. This can be a learnable positional encoding updated during network training, or a fixed positional encoding calculated using a sine function, superimposed on the visual embedding of the corresponding image patch. The hierarchical feature interaction refers to inputting the image patch embedding sequence after superimposed positional encoding into a visual Transformer composed of multiple stacked self-attention encoding layers, performing self-attention calculation layer by layer. For example, twelve layers can be stacked. In another optional implementation, a hierarchical visual Transformer variant with a multi-scale downsampling structure can be used to merge image patch embeddings between different levels. This disclosure does not limit the specific number of layers or structure of the visual Transformer. View identifier encoding is used to label the view to which the image patch belongs in the image patch embedding. In implementation, a learnable view identifier vector can be set for each view, superimposed on all image patch embeddings corresponding to that view. For the aforementioned dual-arm collaborative robot with three camera views, three view identifier vectors are set accordingly.

[0049] In terms of language encoding, the language encoding network first performs word segmentation on the operation instructions. The word segmentation method can be byte pair encoding or word segmentation method such as WordPiece. Each word unit is mapped to a word embedding vector and then superimposed with word order position encoding. This word order position encoding can also be a learnable position encoding or a fixed sinusoidal position encoding.

[0050] The input word embedding sequence after positional encoding is a language Transformer composed of multiple stacked self-attention encoding layers, exemplarily up to six layers, used to establish contextual dependencies between instruction units. The length of the word embedding sequence obtained after sub-word segmentation of the operation instruction can be preset with an upper limit, for example, 64 sub-word units. For operation instructions whose length exceeds this upper limit, the first few sub-word units can be truncated for subsequent processing. For operation instructions whose length is less than the upper limit, the sequence can be padded to the upper limit length at the end with a preset padding character corresponding to the embedding vector. In the self-attention calculation of the language Transformer, a padding mask is used to prevent the embedding corresponding to the padding position from participating in the effective establishment of contextual dependencies. The output of the language Transformer is mapped through a linear projection layer to a multimodal embedding space that matches the visual feature sequence. Here, "matching" means that the feature dimensions of the multimodal embedding space are consistent with the feature dimensions of the aforementioned visual embedding space, both being an example of 768 dimensions, so that the obtained instruction features can participate in subsequent joint processing within the same feature space as the visual feature sequence.

[0051] Based on this, a location query sequence is constructed. The initialization method and the principle for determining the number of learnable query vectors as hyperparameters have been described above and will not be repeated here. Regarding the quantitative relationship between this number and the length of the corresponding view's visual feature sequence, taking the aforementioned 224×224 resolution and 16×16 image block size as an example, the length of the visual feature sequence corresponding to each view is 196. According to the aforementioned example values ​​of sixteen to thirty-two, the number of learnable query vectors is significantly less than the length of the visual feature sequence, satisfying the constraint that the number is less than the length of the visual feature sequence. The length of the visual feature sequence corresponding to different views can be the same or different, depending on the image resolution configuration of the visual sensor corresponding to each view. For views with different visual feature sequence lengths, the number of learnable query vectors corresponding to each view can also be different accordingly. The constraint that the number of query vectors is less than the length of the corresponding view's visual feature sequence is valid for each view separately, rather than requiring that the values ​​be uniform across all views.

[0052] View location encoding is used to label the view to which a learnable query vector belongs, and slot location encoding is used to label the query slots corresponding to each learnable query vector within the same view. Both can be learnable vectors, which are superimposed on the corresponding learnable query vectors. It should be understood that the view location encoding here and the view identifier encoding mentioned in the aforementioned visual encoding process apply to two different types of tokens: location query and image patch. They should be set as independent parameter sets and should not share the same embedding table indexed by view.

[0053] The location query sequence after overlaying the location information is input into the multimodal attention network along with the aforementioned visual feature sequence and instruction feature, which is the joint processing process described in the previous embodiments and will not be repeated here. The statement that "the location query sequence of each view is concatenated with the visual feature sequence and instruction feature sequence to form a joint multimodal sequence and input into the multimodal attention network for bidirectional self-attention calculation" is the same calculation as the attention interaction of the location query sequence on the visual feature sequence and instruction feature in the aforementioned joint processing process, and is not an independent calculation module set up separately outside the multimodal attention network.

[0054] This self-attention calculation can employ a conventional scaled dot-product multi-head self-attention approach. This involves uniformly projecting each token in the joint multimodal sequence onto a linear scale, yielding its corresponding query vector, key vector, and value vector. The dot product of the query vector and key vector is scaled by the square root of the feature dimension and then normalized to obtain attention weights. These weights are then used to weight and sum the value vectors corresponding to the respective visual features, resulting in an aggregated representation for each query vector. This aggregated representation serves as the location query representation carrying visual conditional information. Since location queries can access both visual and instruction features simultaneously in this joint sequence, the aggregated weights of each query vector on the visual features are modulated by the instruction semantic information, ensuring that the resulting location query representation reflects the visual relevance of the instruction condition.

[0055] The action generation module is independent of the multimodal attention network. As a downstream network, it takes the position query representation output by the multimodal attention network after the position query sequence is processed by the position query network and the instruction features as input to represent the condition information required to generate the action. It does not directly access the original visual feature sequence, nor does it serve as a query token for self-attention calculation within the multimodal attention network.

[0056] It should be understood that the query token corresponding to the action generation-related features and the query token corresponding to the aforementioned location query sequence are independent of each other in the token set and do not overlap. That is, the same query token does not have both the identity of a location query sequence query slot and an action generation-related feature at the same time. Therefore, the aforementioned configuration of their respective access permissions does not conflict with each other.

[0057] Regarding the specific configuration of the attention mask matrix, in addition to the aforementioned access relationship between the location query sequence and the visual feature sequence and the access restrictions of other tokens to the visual feature sequence, the newly added access relationship is as follows: the row corresponding to the action generation related features is configured to allow access in the column corresponding to the location query sequence and the instruction feature, and to prohibit access in the column corresponding to the original visual feature sequence. The specific method of prohibiting access is the same as the aforementioned method of restricting the access of the action generation module to the visual feature sequence, and will not be repeated here.

[0058] Since the action generation module is independent of the multimodal attention network, the attention mask matrix only needs to configure the access relationships between the visual feature sequence, the location query sequence, and the instruction feature: the query endpoint corresponding to the location query sequence is configured to allow access to the visual feature sequence and the instruction feature, while the query endpoint corresponding to the visual feature sequence is configured to only allow access to itself. In implementations where the multimodal attention network includes multiple self-attention layers, this configuration of access relationships should be applied to each layer. See, for example... Figure 3 As shown, the visual feature sequence, instruction feature, and location query sequence are input into the multimodal attention network, forming an information pathway in which visual information is indirectly transmitted to action generation-related features via the location query sequence.

[0059] In practice, the operation instructions, rewrite instructions, and counterfactual instructions used for instruction comparison learning correspond to the same set of multi-view visual data collected at the same time. Each of these three instructions is combined with the multi-view visual data and input into the multimodal attention network, resulting in a set of position query representations. The only difference between the three is the input instruction; the visual data variable remains consistent. Here, "same scenario" refers to a physical environment state where the robot body, the object to be operated, and surrounding objects are in the same spatial positional relationship corresponding to the same visual data acquisition, rather than referring to the same type of operation task in general. Two visual data acquisitions at different times, with similar object categories and general layouts but different specific spatial positions, do not fall under the "same scenario" described in this disclosure. The specific construction methods of the instruction variants, the dimensions of the action vectors, and the meanings of each component in the aforementioned embodiments have been described above and will not be repeated here.

[0060] Global pooling can be implemented by taking the arithmetic mean of the representations corresponding to each query slot in the location query sequence, or by setting an additional query slot in the location query sequence that does not participate in visual aggregation and is dedicated to carrying the global representation. The projection network can use a structure of two linear layers with a non-linear activation function (such as GELU) layer in between, mapping the location query representation to a discriminative space suitable for calculating similarity. This space can have the same dimension as the feature dimension of the location query representation itself, or it can be lower; for example, it can be set to 256 dimensions. Regarding the specific calculation of the InfoNCE contrastive loss, let's say the representations of the operation instruction, rewritten instruction, and counterfactual instruction obtained after global pooling and projection network mapping are respectively... The similarity between the two can be expressed using cosine similarity. The global comparison loss can be expressed as: Where τ is a pre-set temperature coefficient hyperparameter, which can be set to 0.1 as an example; preferably, the negative sample set { The representation consists only of counterfactual operation instructions pointing to different operable objects in the same scene, so as to force the location query to focus on the instruction discrimination within the current scene and avoid relying on low-level visual differences across scenes. When the number of counterfactual samples in a single scene is limited, the location query representations corresponding to other scene samples in the same training batch can also be introduced as supplementary negative samples.

[0061] The slot-level InfoNCE contrastive loss is calculated based on the individual representation of each query slot in the location query sequence, rather than the globally pooled representation. For the i-th query slot, the representation of that slot is extracted under three conditions: operation instruction, rewrite instruction, and counterfactual instruction. , according to The same calculation method is used to obtain the loss corresponding to the slot. The slot-level comparison loss is obtained by summing or weighted summing the losses corresponding to each queried slot.

[0062] Taking the aforementioned example of "placing the workpiece in the temporary storage box into the turnover box," if one query slot primarily focuses on the image region containing the temporary storage box and the workpiece during self-attention aggregation, while another query slot primarily focuses on the region containing the turnover box, then the slot-level contrast loss ensures that the representation of the former query slot under the counterfactual instruction "take the labeling tool from the side of the workbench" maintains sufficient distinguishability from its representation under the original operation instruction. This allows different query slots to capture fine-grained spatial evidence related to the instruction. The instruction-conditionalized feature learning loss is obtained by weighting the global level loss and the slot-level contrast loss according to preset weights. The weight values ​​can be determined through hyperparameter search based on the training effect. The acquisition of offset visual data involves the actual execution of at least one preceding skill. Taking the aforementioned production line sorting scenario as an example, if the current target skill to be trained is "taking the labeling tool from one side of the workbench" and the preceding skill is "placing the workpiece in the temporary storage box into the turnover box", then the three cameras can first collect a set of multi-view visual data as the raw visual data in a scenario where several workpieces are placed in the temporary storage box and the turnover box is kept empty. Then, the preceding skill is actually executed. This process brings about the pose change of the robotic arm end effector moving from above the temporary storage box to above the turnover box, the change of the gripper's state from gripping to releasing, and the change of the scene object's spatial state as the workpiece is transferred from the temporary storage box to the turnover box. After the preceding skill is completed and the robotic arm returns to the preset return configuration, the three cameras collect another set of multi-view visual data as the offset visual data.

[0063] The number of preceding skills is not limited to one. When the long-term operation skill chain is long, two or more preceding skills can be executed sequentially before collecting offset visual data. In this case, the scene changes caused by multiple preceding skills are superimposed simultaneously. It should be noted that when the original visual data and the offset visual data are used to calculate the offset consistency loss, the operation instructions are the same. The only difference between the two is whether the visual data itself has undergone scene changes caused by preceding skills. The instruction itself remains unchanged, thus limiting the source of difference targeted by the consistency constraint to the visual level.

[0064] Furthermore, the installation position, orientation, and lighting conditions of each camera in the aforementioned visual sensor group can remain unchanged between the two acquisitions of the original visual data and the offset visual data. That is, apart from the changes in robot end-effector pose, the state of the gripped object, and the spatial state of the scene objects caused by the execution of the preceding skill, the other acquisition conditions do not change, so that the difference between the visual data obtained from the two acquisitions can be attributed to the execution of the preceding skill itself.

[0065] The shared-parameter visual encoding network and the multimodal attention network refer to the two forward computations corresponding to the original visual data and the offset visual data respectively calling the same set of network parameter instances, rather than two sets of networks with the same structure but independent parameters. Similarly, the parameter-sharing projection head can be the same as the projection network involved in the aforementioned instruction contrast learning in terms of network structure, that is, both adopt a structure of two linear layers with nonlinear activation functions such as GELU superimposed in the middle. In terms of parameter configuration, the two can be two independent network instances, which are dedicated to serving the discrimination space of contrast loss and offset consistency loss respectively, or they can share the same set of parameters in another optional implementation. The specific configuration method can be determined according to factors such as training stability, and this disclosure does not limit it.

[0066] It should be noted that the number of time steps in the expert demonstration action trajectory is determined by the teaching duration of the corresponding single-step skill demonstration, and is not required to be equal to the number of time steps in the target action sequence and counterfactual action sequence generated in the inference phase based on the preset integration step size; the time state interpolation on which the flow matching action generation loss is based is constructed within the same expert demonstration action trajectory, and is independent of the specific setting of the integration step size in the inference phase.

[0067] The network parameter values ​​called in the two forward computations of the aforementioned shared parameters remain consistent within the same training iteration; if the visual encoding network or the multimodal attention network contains random processing layers such as random deactivation, the same random state can be fixed in the two forward computations to avoid the introduction of additional differences unrelated to scene offset by such random processing layers, so that the source of difference constrained by the offset consistency loss is limited as much as possible to the scene difference between the original visual data and the offset visual data itself.

[0068] The first position query represents the original visual data path, which serves as the stopping gradient target branch; the second position query represents the offset visual data path, which serves as the gradient update branch. Only the gradient of the offset visual data path is backpropagated, while the representation of the original visual data path is used as a fixed reference target and does not participate in gradient update. This configuration is common in self-supervised representation learning and can, to some extent, prevent both representation paths from converging towards degeneracy.

[0069] The aforementioned correspondence of having the same query slot means that the position query representations obtained by the two forward calculations correspond one-to-one in terms of query slot number. That is, the representation corresponding to the i-th query slot in the first position query representation and the representation corresponding to the i-th query slot in the second position query representation are comparison objects. This correspondence is determined by the order of the query slots in the position query sequence.

[0070] The cosine similarity loss can be expressed as 1 minus the cosine similarity between the two projection representations u and v corresponding to the same view and the same query slot, i.e., 1. sim(u,v); Aggregates the losses for each query slot, using either an arithmetic average or a weighted sum, to obtain the offset consistency loss. See also: Figure 4 As shown, the original visual data and the offset visual data caused by the execution of the preceding skill are respectively input into the visual encoding network with shared parameters and the multimodal attention network to obtain the first position query representation and the second position query representation respectively. In the figure, the connecting line with the blocking mark represents the stopping gradient branch corresponding to the first position query representation, and the connecting line with the arrow represents the gradient update branch corresponding to the second position query representation. The two are mapped by the parameter-shared projection head to calculate the cosine similarity loss.

[0071] The expert demonstration action trajectory, referred to in the flow matching action generation mechanism, is a teaching label used to supervise the action generation network during the training phase. It originates from the single-step skill demonstration data described in the preceding embodiments and is composed of action vectors at each time step, denoted as a*. It should be noted that the expert demonstration action trajectory here is not the same concept as the target action sequence referred to in the inference phase. That is, the former is existing teaching data from the training phase, while the latter is the result predicted by the trained action generation module based on the target operation instruction during the inference phase. The two are at different stages and have different data sources. The random noise action trajectory ε can be sampled from a standard normal distribution, and its dimension and number of time steps are consistent with a. The continuous-time interpolation state can be constructed by linear interpolation between ε and a based on the sampling time t (ranging from 0 to 1). Correspondingly, the target velocity field can be expressed as When using this linear interpolation method, the velocity field does not change with the value of t in the same sampling. The sampling time t can be randomly sampled from a uniform distribution of 0 to 1 in each forward calculation.

[0072] The current position query representation, the instruction characteristics corresponding to the current operation instruction, and the continuous time interpolation state are included. The time encoding corresponding to t is input into the action generation network. The method of obtaining the position query representation and instruction features has been described above and will not be repeated here. The time encoding can be sinusoidal position encoding or a learnable temporal embedding network can be used to map the scalar t into a vector that matches the network input dimension.

[0073] The action generation network is the specific network implementation of the action generation module in the stream matching implementation; the two represent the same technical feature at different levels of expression. This network predicts based on the aforementioned input. Corresponding velocity field ,use and The squared error between the two is used to construct the flow matching action generation loss. By jointly optimizing the prediction error under different values ​​of t, a conditional action trajectory generation model is obtained.

[0074] After the model is trained, the specific method for generating action sequences during the inference phase is as follows: An initial noisy action trajectory is sampled from a standard normal distribution. The sampling time t is gradually increased from 0 to 1 with a preset step size. At each value, the current trajectory state, position query representation, command features, and corresponding time code are input into the model to obtain the predicted velocity field at that time. The current trajectory state is then updated based on the predicted velocity field using a numerical integration method (such as the Euler method). This process is repeated until t reaches 1. The resulting trajectory is the action sequence generated under the corresponding position query representation and command features. The target action sequence and counterfactual action sequence referred to in this scheme are generated in the above manner, respectively, based on the position query representation and command features corresponding to the target operation command and the counterfactual operation command pointing to the interference object. The numerical integration step size and specific integration method can be determined according to the need to balance the accuracy of action generation with the computational time consumption; this disclosure does not limit this.

[0075] Furthermore, the specific construction methods of the difference vector and motion correction vector, and the numerical examples of their dimension-by-dimensional addition with the final motion vector, have been described above and will not be repeated. Regarding the term "linear combination," it should be noted that this combination method is not limited to directly adding the target motion vector and the motion correction vector with the same coefficient. In another optional implementation, a preset attenuation coefficient can be superimposed on the target motion vector itself before adding it to the motion correction vector. The specific coefficient configuration can be determined based on the debugging results of the robotic arm's motion smoothness. The steering weight can be a single value uniformly applied to each time step of the counterfactual motion difference sequence during the same inference process, or different values ​​can be set for different time steps. For each component of the motion vector, the steering weight can also be a scalar uniformly applied to the translation component, attitude component, and gripper opening / closing component. In another optional implementation, it can be set as a weight vector with the same dimension as the motion vector, assigning different weight values ​​to different types of motion components. For example, relatively high weights can be assigned to the translation and attitude components, and relatively low weights to the gripper opening / closing component. The specific weight can be determined based on the gripping tolerance requirements of the object to be operated on.

[0076] The method for determining multiple candidate interference objects is a direct extension of the interference object determination method described previously: after the target detection network completes the detection of the current visual data, if there are two or more foreground objects in the detection results that are different from the operation object corresponding to the target operation instruction, they are respectively identified as candidate interference objects, not limited to the case of a single interference object in the previous embodiment. Taking the previous production line sorting scenario as an example, if before the target operation instruction "take the labeling tool on one side of the workbench" is executed, the current visual data detects three types of foreground objects: the workpiece in the temporary storage box, the workpiece in the turnover box, and the detection fixture temporarily placed on the workbench. All of these are different from the labeling tool corresponding to the target operation instruction, then these three constitute candidate interference objects, and three counterfactual operation instructions are constructed accordingly.

[0077] The query slot representation corresponding to the candidate interference object is determined in the following way: When the object detection network outputs the category of the candidate interference object, it also provides the detection box range of the candidate interference object in the corresponding view image frame; the visual feature token corresponding to the image block contained within the detection box range is checked with the attention weights of each query slot in the location query sequence obtained in the self-attention calculation mentioned above, and the query slot with the highest sum of attention weights for the image blocks within the range is taken as the query slot corresponding to the candidate interference object. Its output representation at the current inference time is the query slot representation corresponding to the candidate interference object; if there are multiple query slots with similar sums of attention weights for the same detection box range, making it difficult to uniquely determine, the arithmetic mean of the representations of these query slots can be taken as the query slot representation corresponding to the candidate interference object. If the query slots are determined in the aforementioned manner, the sum of attention weights corresponding to the detection bounding boxes of the same query slot and two or more candidate interference objects is the highest. Optionally, the query slot representation can be used as the query slot representation for each of the candidate interference objects, participating in the calculation of their respective relevance weights and candidate difference vector weighted aggregation. The implementation of the weighted aggregation process described in this disclosure is not affected by the number of query slots being less than the number of candidate interference objects. The relevance weight is determined based on the attention weights generated in the self-attention calculation between the aforementioned query slot representation and the instruction features corresponding to the target operation instruction. The attention weights used to determine the relevance weights here, and the attention weights used for the weighted aggregation of visual features by the aforementioned position query sequence, belong to the values ​​of two different blocks corresponding to visual features and instruction features in the attention weight matrix obtained in the same self-attention calculation in the multimodal attention network. The two do not conflict with each other and can be obtained together in the same forward calculation. The instruction features corresponding to the target operation instruction can be the instruction features before the language encoding network outputs and inputs into the multimodal attention network, or the instruction features updated by self-attention calculation within the multimodal attention network. The specific feature can be determined according to the implementation needs.

[0078] The attention weight can be taken directly, or it can be normalized (e.g., softmax normalization) within the range of candidate interference objects to obtain the relevant weight, so that the sum of the relevant weights of each candidate interference object is one. For example, if the original attention weights between the query slot representation and the instruction feature corresponding to the target operation instruction for the three candidate interference objects (temporary storage box workpiece, turnover box workpiece, and inspection fixture) are 0.6, 1.0, and 0.4 respectively, and their sum is 2.0, after normalization, the corresponding relevant weights are 0.3, 0.5, and 0.2 respectively; accordingly, the value at a certain time step in the counterfactual action difference sequence is the result of weighted summation of the values ​​of the candidate difference vectors corresponding to the three candidate interference objects at that time step, with weights of 0.3, 0.5, and 0.2 respectively. The weighted aggregation process involves summing each candidate difference vector according to its corresponding time step and dimension using the aforementioned relevance weights to obtain the counterfactual action difference sequence. The subsequent calculation of this difference sequence for the action correction vector has been described above and will not be repeated here. It can be understood that when there is only one candidate interference object, the weighted aggregation degenerates into the single candidate difference vector itself, consistent with the processing method described above when the number of candidates is not distinguished.

[0079] It should be noted that the negative correlation between the turning weight and the prediction uncertainty measure here represents the inverse functional relationship between their values; although both involve the expression "correlation" with the aforementioned relevance weight of the candidate interference object relative to the current operating scenario, they belong to different technical meanings. The former represents the statistical relationship between the values, while the latter represents the degree of semantic association between the candidate interference object and the operating scenario, and they should not be confused.

[0080] The determination of the steering weights is based on multiple independent forward integrals. The single integration process is the same as the numerical integration method used in the previous flow matching inference stage to generate the action sequence. The only difference is that here, for the same target operation command, the same visual data, the same location query representation, and the same command features, multiple independent initial noise sampling and integration solutions are performed, instead of just one solution. The number of independent samplings is a pre-set hyperparameter; for example, eight times can be used. The larger this number, the more stable the resulting uncertainty measure, but the computation time in the inference stage will increase accordingly. The specific value can be determined based on the tolerance for inference latency in the robot application environment. Regarding the specific calculation of the distribution dispersion, the values ​​of the eight candidate target action trajectories at the same time step and the same action dimension can be considered as a set of samples. The standard deviation of this set of samples is calculated to obtain the dispersion value corresponding to that dimension at that time step. The arithmetic mean of the dispersion values ​​corresponding to all time steps and all action dimensions is taken to obtain a single scalar measure characterizing the uncertainty of the current target operation command prediction, denoted as σ. The negative correlation between the shift weight and this uncertainty measure can be represented as an example of an exponential decay form, the specific function form of which can be determined according to the actual deployment scenario.

[0081] The weighted aggregation mechanism and the steering weight determination mechanism can be used together in the same inference process: first, the candidate difference vectors corresponding to multiple candidate interference objects are weighted and aggregated according to the weighted aggregation mechanism to obtain a counterfactual action difference sequence; then, the steering weight value is determined according to the steering weight determination mechanism, and the difference vectors of each time step in the aggregated difference sequence are weighted to obtain the action correction vector. The subsequent participation of the vector in the construction of the final action sequence has been described above and will not be repeated. The two mechanisms can also be used separately. That is, when there is only a single interference object, the steering weight determined by a preset or other method is combined with the unique candidate difference vector; or in the case of multiple candidate interference objects, a preset fixed steering weight is used without dynamic adjustment based on uncertainty measurement. The specific combination method can be selected according to the complexity of the actual deployment scenario.

[0082] The foregoing has provided a detailed description of a long-range robot operation method based on instruction-conditional feature learning, as disclosed in this disclosure. Those skilled in the art will recognize that the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure. The description of the above embodiments is merely for the purpose of helping to understand the method and core ideas of this disclosure. It should be noted that those skilled in the art can make several improvements and modifications to this disclosure without departing from the principles of this disclosure, and these improvements and modifications also fall within the protection scope of the claims of this disclosure.

Claims

1. A long-range robot operation method based on instruction-conditional feature learning, characterized in that, include: The robot acquires multi-view visual data and corresponding operation instructions collected during the execution of skill sequences, performs visual encoding on the multi-view visual data to obtain visual feature sequences for each view, and performs language encoding on the operation instructions to obtain instruction features. A location query sequence corresponding to each view is constructed, the location query sequence including a learnable query vector and its location information encoding; the location query sequence, visual feature sequence and instruction features are input into a multimodal attention network, and the action generation module is limited by an attention mask to obtain instruction-conditional visual information only through the location query sequence, and direct access to the original visual feature sequence is prohibited; Based on the operation instructions and their variants in the same scene, multi-view visual data, and offset visual data caused by prior skills, the position query representation is trained through contrastive learning and consistency constraints; and the above training is jointly optimized with the action generation loss based on multi-view visual data and expert demonstration action trajectories to optimize each network parameter. Visual data of the interfering object introduced by the preceding skills and the target operation command are acquired. Based on the semantic information corresponding to the interfering object in the visual data, a counterfactual operation command pointing to the interfering object is constructed. The visual data is combined with the target operation command and the counterfactual operation command respectively and input into the multimodal attention network to obtain the corresponding position query representations. Each position query representation and the corresponding command are input into the action generation module. Through the multimodal attention network and the action generation module, in the process of generating the flow matching numerical integration, the target velocity field is guided by extrapolation and integral update based on the difference vector between the predicted velocity fields corresponding to the target operation command and the counterfactual operation command respectively, and the final action sequence for controlling the robot actuator is output.

2. The method according to claim 1, characterized in that, The process of performing visual encoding and language encoding to obtain visual feature sequences and instruction features for each view includes: dividing the image blocks according to the image frames corresponding to each view; mapping each image block to a visual embedding space via linear projection; superimposing two-dimensional position encodings onto the corresponding visual embeddings based on the two-dimensional positions of the image blocks in the corresponding image frames; performing hierarchical feature interaction on the embeddings of each image block within the same view through the multi-head self-attention mechanism of the visual Transformer to obtain a view visual feature sequence containing spatial position information of the image blocks; and superimposing view identifier encodings to distinguish the source of the view for different views; performing sub-word segmentation on the operation instructions and mapping them into word embedding sequences; superimposing word order position encodings into the word embedding sequences; establishing contextual dependencies between different instruction terms through the multi-head self-attention mechanism of the language Transformer; and mapping the output via linear projection to a multimodal embedding space that matches the visual feature sequence.

3. The method according to claim 1, characterized in that, The construction of the location query sequence corresponding to each view includes: For each view, a set of independent learnable query vectors is set, and the number of learnable query vectors is less than the length of the visual feature sequence of the corresponding view; Each learnable query vector is superimposed with the view position code representing the view to which it belongs and the position code representing the query slot, respectively, to obtain the position query sequence; The query sequence of each view location is concatenated with the visual feature sequence and the instruction feature sequence to form a joint multimodal sequence, which is then input into a multimodal attention network for bidirectional self-attention calculation. This allows each query vector to extract key features of the operation target from the visual features based on the instruction semantic information. The visual features are then weighted and aggregated according to the attention weights between each query vector and the visual features to obtain a location query representation that carries visual condition information.

4. The method according to claim 1, characterized in that, The action generation module that limits itself to using attention masks to obtain visual information only via the location query sequence includes: The attention interaction relationships between the visual feature sequence, the location query sequence, and the instruction feature are configured using an attention mask matrix. The query end corresponding to the location query sequence is configured to allow access to the visual feature sequence and the instruction feature. The action generation module is independent of the multimodal attention network and takes the location query representation and instruction feature output by the multimodal attention network after processing the location query sequence as input, without directly accessing the original visual feature sequence.

5. The method according to claim 1, characterized in that, Based on the operation instructions and their variants in the same scenario, a location query representation is trained through comparative learning, including: The instruction variants include a rewrite instruction that points to the same operation object and the same target location as the operation instruction, and a counterfactual operation instruction that points to a different operation object than the operation instruction; the operation instruction, rewrite instruction, and counterfactual operation instruction are respectively input into a multimodal attention network to obtain corresponding location query representations; The position query representations corresponding to the operation instruction and the rewrite instruction are globally pooled, mapped by a projection network, and then the positive sample similarity constraint between them is constructed using InfoNCE contrast loss. The global position query representation corresponding to the counterfactual operation instruction is mapped by the projection network and used as a negative sample in the calculation of InfoNCE contrast loss. For each query slot in the location query sequence, the query representation corresponding to the query slot under the operation instruction, the rewrite instruction and the counterfactual operation instruction is extracted respectively. After being mapped by the projection network, the slot-level InfoNCE contrast loss is used to constrain the similarity of the representation of the same query slot under the operation instruction and the rewrite instruction, as well as the similarity interval with the corresponding representation under the counterfactual operation instruction. The instruction-conditional feature learning loss is obtained by weighted summing of the global InfoNCE contrast loss and the slot-level InfoNCE contrast loss of each query slot.

6. The method according to claim 1, characterized in that, The method for acquiring the offset visual data caused by the preceding skill is as follows: according to the skill sequence corresponding to the robot's long-term operation, at least one preceding skill is executed before the current target operation, so that the changes in the spatial state of objects in the scene generated during the execution of the preceding skill are retained in the current operation scene, and after the execution of the preceding skill is completed and the robotic arm is reset to the preset return configuration, the offset visual data is collected by the visual sensor corresponding to the multi-view visual data. Based on the multi-view visual data and the offset visual data, a position query representation is trained through consistency constraints, including: The multi-view visual data and offset visual data are respectively input into a visual encoding network and a multimodal attention network with shared parameters to obtain a first position query representation and a second position query representation with the same query slot correspondence, and are respectively mapped to the consistency discrimination space through a projection head with shared parameters. The projection result corresponding to the first position query representation is used as the stopping gradient target branch, and the projection result corresponding to the second position query representation is used as the gradient update branch. Cosine similarity loss is used to constrain the two projection representations corresponding to the same view and the same query slot, and the two projection representations are aggregated according to the cosine similarity loss of each query slot to obtain the offset consistency loss.

7. The method according to claim 1, characterized in that, The action generation module employs a flow-matching action generation mechanism. It represents the robot expert demonstration action trajectory as a continuous action trajectory. Starting from a random noise action trajectory with the same dimension as the expert demonstration action trajectory, it constructs a continuous-time interpolation state. Based on the vector difference between the random noise action trajectory and the expert demonstration action trajectory, it constructs a target velocity field corresponding to the time state. The position query representation, the instruction features, the continuous-time interpolation state, and the corresponding time code are input into the action generation network. The action generation network predicts the velocity field corresponding to the continuous-time interpolation state. The flow-matching action generation loss is established using the squared error between the predicted velocity field and the target velocity field. By jointly optimizing the velocity fields under different continuous-time states, a conditional action trajectory generation model is obtained.

8. The method according to claim 1, characterized in that, In the process of generating numerical integrals for flow matching, based on the difference vector between the predicted velocity fields corresponding to the target operation command and the counterfactual operation command, the target velocity field is guided by extrapolation and integral update, outputting the final action sequence for controlling the robot actuator, including: According to the corresponding integration time step and the corresponding action dimension, the action velocity fields corresponding to the target operation command and the counterfactual operation command are paired respectively, and the difference vector between the target action velocity field vector and the counterfactual action velocity field vector corresponding to each integration time step is calculated. The difference vector of each integration time step forms the action velocity field difference sequence. The difference vectors of each integral time step in the motion velocity field difference sequence are weighted according to the steering weight to obtain the velocity field correction vector of the corresponding integral time step. The velocity field correction vectors of each integral time step form a velocity field correction sequence with the same integral time step and motion dimension as the target motion velocity field. The target motion velocity field and the velocity field correction sequence are linearly combined according to the corresponding integration time step to obtain the guided velocity field. The guided velocity field is then numerically integrated according to the preset integration step size to output the final motion sequence.

9. The method according to claim 8, characterized in that, When the acquired visual data contains multiple candidate interference objects, a counterfactual operation instruction pointing to each candidate interference object is constructed. Each counterfactual operation instruction is then combined with the visual data and input into the multimodal attention network and the action generation module to obtain candidate counterfactual action velocity fields corresponding to each candidate interference object. Candidate difference vectors between the target action velocity field and each candidate counterfactual action velocity field are calculated according to the corresponding integral time step and corresponding action dimension. Based on the attention weight between the query slot representation corresponding to each candidate interference object in the multimodal attention network and the instruction feature corresponding to the target operation instruction, the relevance weight of each candidate interference object relative to the current operation scene is determined. Weighted aggregation is performed on each candidate difference vector using the relevance weight to obtain the action velocity field difference sequence.