Long-haul operational task-oriented affordance guided bidirectional latent planning method and system

Abstract

The application discloses a kind of availability guidance bidirectional potential planning methods and systems for long-range operation task, the method includes using pre-trained multimodal embedding model to joint coding of robot current visual observation, natural language task instruction and expert demonstration subgoal;Joint optimization reverse planner and forward evaluator, reverse planner is from target condition and generates potential subgoal sequence from regression, and forward evaluator assesses the accessibility of generated subgoal;Abstract potential subgoal is regarded as dynamic query, and the feature map output by visual backbone network is calculated by multi-head cross attention, and the implicit and dynamic availability heat map and task-related visual features are output, and the conditional denoising diffusion probability model is used to predict continuous action with the condition of grounded visual features and robot body sensation state.The application significantly improves the execution success rate and cross-stage consistency of robot under multi-stage, multi-skill composite task.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and long-range operation planning and control of robots, specifically to an availability-guided bidirectional potential planning method and system for long-range operation tasks. Background Technology

[0002] The ability of robots to autonomously complete complex, multi-step tasks in real-world, open environments is a core requirement for achieving general embodied intelligence. This capability relies on long-horizon manipulation, which involves executing a series of actions sequentially over a considerable time span to achieve a final goal. For example, in household tidying tasks, a robot needs to complete multi-stage tasks sequentially, such as "placing tomatoes on a decorative plate and stacking bowls" or "hanging cups on a cup rack first, then stacking the remaining cups." In industrial assembly scenarios, robots need to complete continuous multi-step operations involving grasping, moving, aligning, and assembling. These types of tasks require the system to possess robustness in two aspects:

[0003] Firstly, temporal robustness, meaning that the generated sub-target sequences maintain logical coherence and dynamic consistency over long time spans;

[0004] The second aspect is spatial robustness, which means reliably grounding abstract high-level planning into a noisy, high-dimensional perception-action space during execution.

[0005] To accomplish such tasks, a layered architecture of "high-level sub-goal planning + low-level strategy execution" has become the mainstream paradigm. However, existing methods often fail to simultaneously meet both requirements, mainly due to the following shortcomings:

[0006] 1) Fine-grained forward prediction method: Although dense video prediction models, represented by Seer, can provide frame-by-frame fine-grained future trajectory guidance, the computational cost is huge and it is difficult to meet the requirements of real-time control.

[0007] 2) Coarse-grained forward prediction method: Methods represented by SuSIE et al. use a larger step size for forward planning, which can improve computational efficiency. However, due to the nature of the autoregressive process, the early prediction error will accumulate and amplify along the time axis, causing the planning to deviate from the final goal, resulting in poor time robustness.

[0008] 3) One-way Latent Backward Planning (LBP): Methods such as LBP deduce a sequence of sub-goals from the final goal, which can alleviate the accumulated error of forward methods to some extent. However: First, one-way backward planning lacks explicit constraints on the reachability of the current state, which may result in sub-goals that are inconsistent with or unreachable from the current state. Second, even if the resulting sub-goals are semantically reasonable, the underlying strategy may still fail to correctly ground the abstract intention to the specific visual area due to visual ambiguity or semantic interference, resulting in execution failure, because of the "planning-execution gap" between the latent space and the high-dimensional perception-action space. For example, in the task of "putting two moka pots on the stove," the backward planner's output "grab the first moka pot" is semantically valid, but the underlying strategy may fail to identify the correct handle grab point due to visual ambiguity. This is essentially not a time problem but a space problem, highlighting the necessity of perceptual grounding.

[0009] In summary, existing methods have the following shortcomings: First, they lack a unified planning mechanism that simultaneously introduces the dual constraints of "goal orientation" and "dynamic consistency" during the training phase; second, they lack a structured mechanism that dynamically grounds abstract potential sub-goals to the current visual perception space; third, in a hierarchical architecture, high-level planning and low-level execution usually need to be designed separately, making it difficult to perform end-to-end joint optimization with a single goal; and fourth, they lack a reproducible experimental framework for verifying feasibility and robustness on long-term, multi-stage real-world tasks.

[0010] To address the aforementioned challenges, this invention proposes an availability-guided bidirectional latent planning method and system for long-range operational tasks. This invention utilizes a collaborative mechanism of "bidirectional latent planning—availability grounding—diffusion action prediction" to achieve temporal robustness through bidirectional training, spatial robustness through availability grounding, and end-to-end joint optimization through single action loss. This significantly improves the robot's execution success rate and cross-stage consistency in long-range, multi-stage operational tasks. Summary of the Invention

[0011] Purpose of the invention: This invention proposes an availability-guided bidirectional latent planning method and system for long-range operation tasks, which solves the problems of existing methods in long-range operation tasks, such as the difficulty in simultaneously satisfying the binary constraints of "time robustness-space robustness", tight coupling between planning and execution and the inability to perform end-to-end joint optimization, thereby improving the robustness and executability of robots in long-range multi-stage operation tasks.

[0012] Technical solution: The present invention provides an availability-guided bidirectional latent planning method for long-range operation tasks, comprising the following steps:

[0013] (1) Shared latent space encoding: A pre-trained multimodal embedding model is used to encode the robot's current visual observations. Natural Language Task Instructions The initial state potential representation is obtained by jointly encoding the expert demonstration sub-objectives. Target potential representation With expert sub-target sequence ;

[0014] (2) Optimize the bidirectional latent planner: jointly optimize the backward planner within the shared latent space. With forward evaluator The reverse planner is... and To generate conditionally autoregressively, the length is... sub-target sequences The forward evaluator evaluates the reachability of each sub-goal generated by the backward planner and feeds back the result as a supervision signal to the backward planner.

[0015] (3) Availability grounding: For the current observation in the inference phase The sequence of potential sub-objectives output by the reverse planner Potential representation of the initial state The task query is obtained after fusion and projection. The task query is used as input to perform multi-head cross-attention calculation on the visual feature sequence output by the visual backbone network, and the attention weights are output. Task-related grounding visual features ;

[0016] (4) Strategy decoding and execution: The grounding visual features With respect to the robot's current proprioceptive state The concatenation is used as conditional input to the policy decoder. The policy decoder uses a conditional denoising diffusion probability model to predict continuous actions. And then assign the task to the robot to execute;

[0017] (5) Two-stage training strategy: First train the bidirectional latent planner, and then predict the loss with a single action. End-to-end joint optimization of the availability grounding module and the policy decoder is performed to ensure the correlation between the visual characteristics of grounding and the final action generation.

[0018] Furthermore, the shared latent space described in step (1) consists of a pre-trained visual-language contrastive embedding model, including a DecisionNCE encoder, which encodes visual observations, language instructions, and expert demonstration sub-goals, so that the three are located in a unified latent space, supporting the joint training and cross-modal reasoning of the backward planner, forward evaluator, and availability grounding module in the same space.

[0019] Furthermore, the reverse planner described in step (2) Including the initial predictor With recursive Transformer decoder The autoregressive generation method is as follows:

[0020]

[0021]

[0022] The initial predictor first predicts the sub-target that is furthest from the current state. Then, the recursive Transformer decoder takes historical predictions as the target input and... Input to memory, generated sequentially and regressively. This forms a reverse planning process from far to near;

[0023] The forward evaluator For a multilayer perceptron, from any candidate sub-target Compared with the initial state Potential characteristics of the predicted final goal:

[0024] .

[0025] Furthermore, the reverse planner in step (2) uses reverse imitation loss. Learning by imitation:

[0026]

[0027] in, For cosine similarity loss, For the first Potential representations of expert sub-objectives;

[0028] The forward evaluator For a multilayer perceptron, from any candidate sub-target Compared with the initial state Potential representations of the final target from the starting point:

[0029]

[0030] The training loss of the forward evaluator itself In the expert sub-target Above calculation:

[0031]

[0032] The forward prediction loss is calculated by examining the sub-objectives generated by the backward planner. Apply dynamic consistency constraints to the reverse planner:

[0033] ;

[0034] The overall objective function of the bidirectional latent planner Defined as the weighted sum of the reverse imitation loss and the forward consistency loss:

[0035]

[0036] in, The consistency weight coefficient is used; by minimizing the overall objective function, the reachability and dynamic consistency constraints learned by the forward evaluator are distilled into the parameters of the backward planner, so that in the inference phase, only the backward planner is used to generate a sequence of sub-objectives that are both goal-oriented and dynamically consistent.

[0037] Further, the task query in step (3) The construction method is as follows: using the initial state as the latent representation As a query, the sub-objective sequence output by the reverse planner. As a key value, it is used by the target fusion module. The state-aware vector is obtained by performing cross-attention fusion:

[0038]

[0039] Then through the linear projection layer Mapping to visual feature dimensions ,get ,in, Batch size; the visual backbone network will be used for current observations Feature map obtained by encoding Flattened into a sequence along spatial dimensions As a key-value pair.

[0040] Further, the multi-head cross-attention calculation in step (3) is defined as:

[0041]

[0042] Attention weight After being reshaped in the spatial dimension, it forms an implicit and dynamic availability heatmap. Identify the image region in the current observation that is most relevant to the task query; availability-weighted visual evidence. After further processing using a standard Transformer block, task-related grounding visual features are obtained:

[0043]

[0044] Grounding visual characteristics As an information bottleneck, only visual features relevant to the current task are retained.

[0045] Further, the policy decoder described in step (4) A conditional denoising diffusion probability model is used to concatenate vectors. As a condition, from the actions of experts with noise Predicting Gaussian noise Action prediction loss Defined as:

[0046]

[0047] in, For the time step of the diffusion process, For parameterized noise prediction networks, For robots at all times The state of proprioception.

[0048] Furthermore, the two-stage training strategy described in step (5) is as follows:

[0049] In the first phase, only the bidirectional latent planner is trained by minimizing the overall objective function. The backward planner and the forward evaluator are jointly optimized; after training, the parameters of the backward planner are frozen, and the forward evaluator is discarded during the inference phase.

[0050] In the second stage, end-to-end joint training is performed. The inverse planner obtained in the first stage is fixed, and the availability grounding module and policy decoder are jointly optimized end-to-end. During the optimization process, loss is predicted using a single action. Backpropagation allows gradients to flow simultaneously to the cross-attention module, the projection layer, and the noise prediction network, enabling the availability representation to learn adaptively in the direction of minimizing the action prediction error.

[0051] The present invention discloses an availability-guided bidirectional latent planning system for long-range operational tasks, comprising:

[0052] Shared latent space encoding module: Based on a pre-trained vision-language embedding model, it encodes visual observations, language instructions, and expert sub-objectives, and outputs state latent representations, target latent representations, and expert sub-objective sequences;

[0053] The bidirectional latent planning module includes a backward planner and a forward evaluator. The forward evaluator evaluates the reachability of each sub-objective generated by the backward planner and feeds back the result as a supervision signal to the backward planner; and jointly trains it.

[0054] Availability grounding module: includes target fusion module, projection layer, visual backbone network and multi-head cross attention layer, outputs availability heatmap and grounding visual features;

[0055] Policy Decoder: Employs a conditional denoising diffusion probability model to predict robot actions based on grounded visual features and proprioceptive state.

[0056] Furthermore, the system also includes a two-stage training module: first, a bidirectional latent planner is trained, and then the loss is predicted using a single action. End-to-end joint optimization of the availability grounding module and the policy decoder is performed to ensure the correlation between the visual characteristics of grounding and the final action generation.

[0057] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0058] 1. This invention proposes a "bidirectional latent planning" mechanism, which distills dynamic consistency constraints into the parameters of the reverse planner in the form of a supervision signal through joint training of the reverse planner and the forward evaluator. This allows the generation of sub-objective sequences that are both goal-oriented and dynamically consistent without calling the forward evaluator during the inference phase. Compared with the unidirectional reverse planning method, this significantly improves the time robustness of long-term tasks without increasing any computational overhead during inference.

[0059] 2. This invention proposes an "availability grounding" mechanism, which uses abstract potential sub-targets as dynamic queries to perform multi-head cross-attention operations on visual feature maps, outputting implicit availability heatmaps and task-related grounded visual features. This decouples the complex visual motion problem into two sub-problems: "where to look" and "how to do it," significantly improving spatial robustness and the ability to resist visual and semantic interference.

[0060] 3. The present invention adopts an end-to-end joint optimization strategy, which uses only the single action prediction loss to jointly train the availability grounding module and the policy decoder, so that the availability representation adaptively learns in the direction most helpful to the final action generation, which significantly improves the synergistic effect between modules compared with staged separate training.

[0061] 4. This invention achieved an average success rate of 97.1% on 10 multi-stage tasks of the LIBERO-LONG long-range simulation benchmark, which is 8.5% higher than the previous best method, and 7 out of the 10 tasks achieved a perfect success rate of 100%.

[0062] 5. This invention has verified the transferability from simulation to reality on four long-term tasks (including 4-stage and 6-stage tasks) on a real robot platform (xArm6 + dual RealSense D435 cameras). In particular, this invention can still maintain a stable execution success rate in the later stages of the 6-stage task (when the baseline method only performs at 5% and 2%). Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0064] Figure 2 This is a schematic diagram of 10 long-range operation tasks of the present invention on the LIBERO-LONG simulation benchmark;

[0065] Figure 3 This is a schematic diagram of the ablation experiment results of the reverse programming-forward evaluation joint training mechanism and the availability grounding mechanism on the LIBERO-LONG benchmark of the present invention;

[0066] Figure 4 This is a visual diagram illustrating the experimental setup of the present invention and the multi-stage execution of four real long-range operation tasks.

[0067] Figure 5 This is a schematic diagram comparing the stage-by-stage average score of the present invention on four real-world long-range operational tasks with the baseline method LBP.

[0068] Figure 6 This is a visualization diagram of the availability heatmap of the availability grounding mechanism of the present invention under different tasks. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings.

[0070] This invention models long-range language-conditional robot operation as a context-conditional visual motion policy learning problem. Receive current visual observation With context Output underlying actions Context It can consist of multiple modalities: natural language task instructions Provide task identity, target image Provide expected results and potential sub-objectives Provides task progress. This invention utilizes expert demonstration trajectory datasets. The above training strategy is used for each trajectory. It consists of a series of observation-action pairs. The length of the trajectory.

[0071] The goal of policy learning is to maximize the log-likelihood of expert actions given the context:

[0072]

[0073] Among them, expectations In context distribution The upper-valued, contextual distribution models various task-related signals available for a given trajectory. For example, This is for cases where only language instructions are used; This is for situations where target images and language commands are used simultaneously. Furthermore, future sub-goals are introduced as a joint context. This formalization enables the invention to flexibly combine multiple guidance signals within a unified visual motion strategy and dynamically switch behavioral granularity between abstract task intent and concrete continuous control.

[0074] Therefore, an availability-guided bidirectional latent programming method for long-range operational tasks is proposed, consisting of four core architectures (AGiLe framework), such as... Figure 1 As shown, the framework includes shared latent space encoding, bidirectional latent planning, availability grounding, and policy decoding. Space encoding and bidirectional latent planning together constitute the "high-level planning" part, while availability grounding and policy decoding together constitute the "low-level execution" part. The framework employs a two-stage training strategy: the first stage trains only the bidirectional latent planning module; the second stage freezes the backpropagator parameters and performs end-to-end joint optimization of the availability grounding module and the policy decoder. The specific implementation process is as follows:

[0075] (1) Shared latent space encoding.

[0076] To enable joint training and cross-modal inference of the backward planner, forward evaluator, and availability grounding module within a unified feature space, this invention employs a pre-trained visual-language contrastive embedding model (in a specific embodiment, the RN50-CLIP encoder of DecisionNCE) as a shared latent space provider. The encoder remains frozen during both training and inference. Specifically: the current visual observation... The initial state latent representation is obtained after encoding. Natural Language Task Instructions The target latent representation is obtained after encoding. ;

[0077] The keyframe sub-targets demonstrated by the expert were obtained after encoding. ,in The number of sub-targets.

[0078] Since all potential representations reside in the same space, the reverse planner is able to operate within this unified space. and For conditional regression The forward evaluator can directly match candidate sub-targets with... Perform an accessibility comparison.

[0079] (2) Optimize the bidirectional potential planner.

[0080] Bidirectional latent programming aims to generate a set of reasonable and consistent abstract sub-goal plans. This invention employs a reverse planner. With forward evaluator A two-way mechanism for joint training.

[0081] The reverse planner consists of an initial predictor. With recursive Transformer decoder The process involves generating a sequence of sub-targets autoregressively. First, the initial predictor predicts the sub-target furthest from the current state (i.e., closest to the target):

[0082]

[0083] The recursive Transformer decoder then uses historical predictions. As the target input (tgt), with As memory input, subsequent sub-goals are generated step by step:

[0084]

[0085] This forms a backward planning process from far to near. The backward planner is trained using imitation learning, minimizing the backward imitation loss between the predicted sub-objective and the expert sub-objective:

[0086]

[0087] in, This represents the cosine similarity loss. For the first Potential representations of expert sub-objectives.

[0088] To ensure the consistency of the generated sub-objectives in terms of dynamics and the reachability of the target, this invention introduces a forward evaluator. It is implemented as a multilayer perceptron with layer normalization, and its function is to select from any candidate sub-targets. Compared with the initial state Potential representations of the final target from the starting point:

[0089]

[0090] The forward evaluator is trained using a forward consistency loss on expert sub-objectives:

[0091]

[0092] The forward evaluator evaluates the sub-goals generated by the backward planner. Calculate the forward prediction loss and apply the dynamic consistency constraint to the backward planner:

[0093]

[0094] The overall objective function of bidirectional latent programming is a weighted sum of the backward imitation loss and the forward consistency loss:

[0095]

[0096] in, For consistency weighting coefficients. The overall objective function also forces the reverse planner to: Semantically close to expert demonstrations; through It satisfies the dynamic reachability constraints identified by the forward evaluator.

[0097] The "bidirectional" mechanism proposed in this invention is mainly reflected in the training phase. By jointly optimizing the backward planner and the forward evaluator, the reachability and dynamic consistency knowledge learned by the forward evaluator is effectively "distilled" into the parameters of the backward planner. This is achieved by forcing the backward planner to implicitly satisfy the verification constraints of the forward evaluator (i.e., minimizing...). The reverse planner learns to internalize these dynamic constraints. This mechanism allows the present invention to discard the forward evaluator directly during the inference phase, achieving time robustness using only the reverse planner without any additional deployment-time computational overhead.

[0098] This process can be understood as a kind of "constrained imitation learning": the reverse planner learns to imitate the main task demonstrated by the expert (through...). The forward evaluator, acting as a learned "goal-alignment teacher," provides auxiliary supervision for dynamic consistency. This bidirectional coupling offers a fundamental improvement over the unidirectional programming architecture.

[0099] (3) Available grounding module.

[0100] High-level sub-objective plans output by the bidirectional potential planning module It exists only in the latent space, lacking a direct connection to the higher-dimensional visual world. For this plan to be reliably executed, the strategy must dynamically ground the abstract intention to the current observation. In the specific visual context, this invention introduces an "affordance as structural guidance" mechanism, which decouples the complex visual motion problem into two co-trained subtasks: "where to look" (affordance grounding) and "how to do it" (action generation).

[0101] Original sub-target sequence It is a set of potential vectors. To compress it into a single state-aware vector, this invention uses a backward planner to encode the initial state. As a query, with the entire sequence As a key value, in the target fusion module Perform cross-attention fusion:

[0102]

[0103] Subsequently, by the linear projection layer Will Projected onto visual feature dimension , obtain the final task query ,in For batch size.

[0104] Visual backbone network of strategy (In one specific embodiment, a pre-trained and fine-tuned SwinTransformer) for the current observation Encoding is performed to obtain a feature map that preserves the spatial grid correspondence. ,in and This represents the spatial height and width of the feature map. The feature map is then flattened along the spatial dimensions into a sequence:

[0105]

[0106] As a key of the grounding mechanism AND value .

[0107] This invention explicitly uses multi-head cross-attention to compute the alignment between task intent and spatial visual context:

[0108]

[0109] This single operation produces two key outputs: (i) attention weights After being reshaped in the spatial dimension, an implicit and dynamic availability heatmap M is formed, identifying the image regions most relevant to the current task query in the current observation; (ii) availability-weighted visual evidence Further processing via a standard Transformer block (feedforward network + residual connections + layer normalization):

[0110]

[0111] Obtain task-related grounding visual features This vector acts as an information bottleneck—a "refined" visual representation that contains only visual features relevant to the current task and is correlated with the robot's proprioceptive state. Together they form the complete ground input of the final policy decoder.

[0112] (4) Strategy decoding (conditional denoising diffusion probability model).

[0113] The availability grounding module provides task-related visual features for the subsequent policy decoder. The ultimate goal of this invention is to train a policy decoder. Based on this purification context and the robot's proprioceptive state Generate correct continuous actions This invention implements the policy decoder as a conditional denoising diffusion probabilistic model (DDPM). This model uses concatenated vectors... As a condition, learn from the actions of experts with noise. Predicting Gaussian noise ,in, This represents the time step in the diffusion process.

[0114] The key feature of this invention lies in its end-to-end optimization strategy. In the second phase of training, this invention predicts the loss using a single, uniform action. For available grounding modules ( ) and policy decoder ( End-to-end joint optimization is performed. Specifically, this invention minimizes the standard DDPM objective, i.e., prediction noise. Compared with real Gaussian noise Mean square error between:

[0115]

[0116] This single loss provides a unified monitoring signal: from The gradients are simultaneously backpropagated to both the policy decoder and the availability grounding module. This mechanism not only updates the policy decoder to improve action generation based on given features, but also updates the availability grounding module (i.e., the cross-attention layer and the projection layer) to adjust the attention weights. This allows it to learn adaptively in the direction of minimizing the final action error, thus ensuring that grounding is task-dependent.

[0117] This invention also proposes an availability-guided bidirectional latent programming system for long-range operational tasks, comprising:

[0118] Shared latent space encoding module: Based on a pre-trained vision-language embedding model, it encodes visual observations, language instructions and expert sub-objectives, and outputs state latent representations, target latent representations and expert sub-objective sequences.

[0119] The bidirectional latent planning module includes a backward planner and a forward evaluator. The forward evaluator evaluates the reachability of each sub-objective generated by the backward planner and feeds back the result as a supervision signal to the backward planner; and jointly trains it.

[0120] Availability grounding module: includes target fusion module, projection layer, visual backbone network and multi-head cross attention layer, outputs availability heatmap and grounding visual features.

[0121] Policy Decoder: Employs a conditional denoising diffusion probability model to predict robot actions based on grounded visual features and proprioceptive state.

[0122] Two-stage training module: First, train a bidirectional latent planner, then predict the loss using a single action. End-to-end joint optimization of the availability grounding module and the policy decoder is performed to ensure the correlation between the visual characteristics of grounding and the final action generation.

[0123] like Figure 2 As shown, the typical workflow of this invention is as follows: the system receives the user's natural language task instructions. (e.g., "Place tomatoes on a decorative plate and stack bowls") and current observations The shared latent space coding module will and They are coded as follows and Reverse planner by and Given the condition, sub-target sequences are generated autoregressively. (The forward evaluator is no longer called during the inference phase); the availability grounding module is... For query, Task query for key values Visual backbone coding is currently observed. Obtain visual feature sequence The two generate an availability heatmap in multi-headed cross-attention. visual characteristics of grounding The policy decoder uses To predict continuous actions using a conditional denoising diffusion probability model. Robot execution And collect new observations The process continues until the task is completed or the maximum number of execution steps is reached.

[0124] In this embodiment, the network architecture and training hyperparameters of each module are as follows:

[0125] Bidirectional latent programming module: Initial predictor With forward evaluator Shared symmetric multilayer perceptron architecture (including layer normalization); recursive planner It is implemented as a multi-layer Transformer decoder (including multi-head attention).

[0126] Available grounding module: visual backbone A pre-trained Swin Transformer was used and fine-tuned in the second stage; the target fusion module... Implemented as a multi-layer cross-attention Transformer; grounded projection It is implemented as a linear layer; the core grounding module is a standard multi-head cross-attention layer.

[0127] Policy Decoder: It is implemented as a multi-layer residual sensor, using spliced ​​grounding characteristics and proprioceptive state. As a condition.

[0128] Training: The AdamW optimizer, cosine annealing learning rate scheduling, and linear warm-up phase were used; training was conducted on two NVIDIA A5000 GPUs in a distributed data parallel (DDP) manner.

[0129] To verify the effectiveness of this invention in long-range operation tasks, it was compared with several mainstream methods on the LIBERO-LONG benchmark. This benchmark includes 10 multi-stage operation tasks, such as... Figure 2 As shown, each task was trained using 50 expert demonstrations.

[0130] This invention achieved an average success rate of 97.1% on LIBERO-LONG, an 8.5% improvement over the previous best method, LBP. In 7 out of 10 tasks, this invention achieved a perfect success rate, while LBP and Seer only achieved this level in 3 and 1 tasks, respectively. Even in the most challenging tasks (such as "placing both pots on the stove" and "placing soup and sauce in a basket"), this invention demonstrated significantly better performance than other methods.

[0131] To evaluate the contribution of each core component of this invention, ablation experiments were conducted on the LIBERO-LONG benchmark, and the results are as follows: Figure 3 As shown.

[0132] The role of bidirectional latent programming: Constructing a variant "AGiLe (without a forward evaluator)" that incorporates consistency weights Set to 0, only use reverse imitation loss Train the reverse planner and remove the forward validation flow. and The results showed that the average success rate of this variant decreased by 8.0% (from 97.1% to 89.0%), and the number of perfectly completed tasks decreased from 7 to 3. This confirms the crucial role of bidirectional design in the time robustness of long-term planning: the forward evaluator acts as a consistency regulator, filtering out dynamically inconsistent sub-goals and ensuring that the generated trajectories are time-coherent, goal-aligned, and executable.

[0133] The role of availability grounding: Constructing the variant "AGiLe (without availability grounding)" replaces the multi-head cross-attention module with a global average pooling and fusion planning vector of visual backbone features. and status The simple splicing method was used. The average success rate of this variant dropped from 97.1% to 90.5%, and it failed to maintain perfect performance on multiple tasks. This confirms the crucial role of availability grounding in precise operation: it guides the strategy to focus on task-relevant areas, suppresses irrelevant semantic and visual interference, and improves spatial robustness.

[0134] In summary, the two core components complement each other: the bidirectional planner ensures temporal robustness by generating logically coherent and goal-consistent sequences of sub-objectives; the availability grounding module ensures spatial robustness by grounding abstract sub-objectives into executable actions. Together, they support the superior performance of this invention in long-range operational tasks.

[0135] To verify the portability of this invention from simulation to reality, in one specific embodiment, the system was deployed on a real robot platform. The hardware configuration included an xArm6 six-DOF robotic arm and two Intel RealSense D435 RGB-D cameras (one wrist-mounted and one top-down view), as follows. Figure 4 As shown in the middle left figure, four challenging long-range operational tasks are constructed, such as... Figure 4 As shown in the middle right figure, two are 4-stage tasks and two are 6-stage tasks, designed to evaluate the model's subtask planning and execution capabilities under complex multi-step language instructions. The specific task instructions are:

[0136] Task 1 (Phase 4): "Place tomatoes on a decorative plate and stack bowls";

[0137] Task 2 (Stage 4): "Place the peaches on the white plate and stack the bowls";

[0138] Task 3 (Stage 6): "Hang the cups on the cup rack first, then stack the remaining cups";

[0139] Task 4 (Stage 6): "Stack three cups together, then hang the remaining cups on the cup rack."

[0140] The four tasks collected 100, 100, 150, and 150 expert demonstrations respectively, covering multi-view observation and continuous control commands.

[0141] A fine-grained percentage-based scoring system is adopted—each stage has a maximum score of 100 points (25 points for correct intent, 50 points for successful capture, 75 points for correct transport, and 100 points for successful placement), and a strict "all or nothing" sequential gating mechanism is used: the next stage can only be entered when the current stage reaches a full 100 points. Each task is executed 10 times (rollout), and the final score is the average of the three most recently saved model checkpoints.

[0142] Figure 5 The average stage-by-stage scores of the present invention and the baseline method LBP on four real-world tasks are presented. The results show that on the 4-stage tasks (Tasks 1 and 2), the present invention significantly outperforms LBP in all stages; on the 6-stage tasks (Tasks 3 and 4), the performance gap widens further—LBP drops sharply in later stages (only 5% in Task 3 and only 2% in Task 4), while the present invention maintains a stable execution success rate. This phenomenon verifies the robustness of the present invention's bidirectional planning and availability grounding mechanism to real-world long-range tasks.

[0143] To further illustrate the working principle of the availability grounding mechanism more intuitively, this invention applies attention weights to multiple simulation tasks. Visualize, such as Figure 6 As shown, the availability heatmap adaptively focuses on the visual region most relevant to the current stage, corresponding to different task instructions: for example, in the task of "putting both kettles on the stove," the heatmap focuses on the kettles and the stove; in the task of "turning on the stove and placing the pot on it," the heatmap first focuses on the stove switch and then on the kettles; in the task of "putting the cup in the microwave and closing the door," the heatmap first focuses on the cup and then on the microwave door. This visualization result qualitatively confirms that the availability representation learned in this invention does indeed dynamically align abstract planning with concrete visual evidence.

[0144] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. An availability-guided bidirectional latent programming method for long-range operational tasks, characterized in that, Includes the following steps: (1) Shared latent space encoding: A pre-trained multimodal embedding model is used to encode the robot's current visual observations. Natural Language Task Instructions The initial state potential representation is obtained by jointly encoding the expert demonstration sub-objectives. Target potential representation With expert sub-target sequence ; (2) Optimize the bidirectional latent planner: jointly optimize the backward planner within the shared latent space. With forward evaluator The reverse planner is... and To generate conditionally autoregressively, the length is... sub-target sequences The forward evaluator evaluates the reachability of each sub-goal generated by the backward planner and feeds back the result as a supervision signal to the backward planner. (3) Availability grounding: For the current observation in the inference phase The sequence of potential sub-objectives output by the reverse planner Potential representation of the initial state The task query is obtained after fusion and projection. The task query is used as input to perform multi-head cross-attention calculation on the visual feature sequence output by the visual backbone network, and the attention weights are output. Task-related grounding visual features ; (4) Strategy decoding and execution: The grounding visual features With respect to the robot's current proprioceptive state The concatenation is used as conditional input to the policy decoder. The policy decoder uses a conditional denoising diffusion probability model to predict continuous actions. And then assign the task to the robot to execute; (5) Two-stage training strategy: First train the bidirectional latent planner, and then predict the loss with a single action. End-to-end joint optimization of the availability grounding module and the policy decoder is performed to ensure the correlation between the visual characteristics of grounding and the final action generation.

2. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, The shared latent space in step (1) consists of a pre-trained visual-language contrastive embedding model, including a DecisionNCE encoder, which encodes visual observations, language instructions and expert demonstration sub-goals, so that the three are located in a unified latent space, supporting the joint training and cross-modal reasoning of the backward planner, forward evaluator and availability grounding module in the same space.

3. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, The reverse planner in step (2) Including the initial predictor With recursive Transformer decoder The autoregressive generation method is as follows: The initial predictor first predicts the sub-target that is furthest from the current state. Then, the recursive Transformer decoder takes historical predictions as the target input and... Input to memory, generated sequentially and regressively. This forms a reverse planning process from far to near; The forward evaluator For a multilayer perceptron, from any candidate sub-target Compared with the initial state Potential representations of the final target from the starting point: 。 4. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, Step (2) The reverse planner uses reverse imitation loss. Learning by imitation: in, For cosine similarity loss, For the first Potential representations of expert sub-objectives; The forward evaluator For a multilayer perceptron, from any candidate sub-target Compared with the initial state Potential representations of the final target from the starting point: The training loss of the forward evaluator itself In the expert sub-target Above calculation: The forward prediction loss is calculated by examining the sub-objectives generated by the backward planner. Apply dynamic consistency constraints to the reverse planner: ； The overall objective function of the bidirectional latent planner Defined as the weighted sum of the reverse imitation loss and the forward consistency loss: in, The consistency weight coefficient is used; by minimizing the overall objective function, the reachability and dynamic consistency constraints learned by the forward evaluator are distilled into the parameters of the backward planner, so that in the inference phase, only the backward planner is used to generate a sequence of sub-objectives that are both goal-oriented and dynamically consistent.

5. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, Step (3) Task query The construction method is as follows: using the initial state as the latent representation As a query, the sub-objective sequence output by the reverse planner. As a key value, it is used by the target fusion module. The state-aware vector is obtained by performing cross-attention fusion: Then through the linear projection layer Mapping to visual feature dimensions ,get ,in, Batch size; the visual backbone network will be used for current observations Feature map obtained by encoding Flattened into a sequence along spatial dimensions As a key-value pair.

6. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, The multi-head cross-attention calculation in step (3) is defined as follows: Attention weight After being reshaped in the spatial dimension, it forms an implicit and dynamic availability heatmap. Identify the image region in the current observation that is most relevant to the task query; availability-weighted visual evidence. After further processing using a standard Transformer block, task-related grounding visual features are obtained: Grounding visual characteristics As an information bottleneck, only visual features relevant to the current task are retained.

7. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, Step (4) Policy Decoder A conditional denoising diffusion probability model is used to concatenate vectors. As a condition, from the actions of experts with noise Predicting Gaussian noise Action prediction loss Defined as: in, For the time step of the diffusion process, For parameterized noise prediction networks, For robots at all times The state of proprioception.

8. The availability-guided bidirectional latent programming method for long-range operation tasks according to claim 1, characterized in that, The two-stage training strategy described in step (5) is as follows: In the first phase, only the bidirectional latent planner is trained by minimizing the overall objective function. Jointly optimize the backward planner and the forward evaluator; After training is complete, the parameters of the backward planner are frozen, and the forward evaluator is discarded during the inference phase. In the second stage, end-to-end joint training is performed. The inverse planner obtained in the first stage is fixed, and the availability grounding module and policy decoder are jointly optimized end-to-end. During the optimization process, loss is predicted using a single action. Backpropagation allows gradients to flow simultaneously to the cross-attention module, the projection layer, and the noise prediction network, enabling the availability representation to learn adaptively in the direction of minimizing the action prediction error.

9. An availability-guided bidirectional latent planning system for long-range operational tasks employing the method described in any one of claims 1 to 8, characterized in that, include: Shared latent space encoding module: Based on a pre-trained vision-language embedding model, it encodes visual observations, language instructions, and expert sub-objectives, and outputs state latent representations, target latent representations, and expert sub-objective sequences; Bidirectional latent planning module: includes a backward planner and a forward evaluator. The forward evaluator evaluates the reachability of each sub-goal generated by the backward planner and feeds back the result as a supervision signal to the backward planner. And conduct joint training with them; Availability grounding module: includes target fusion module, projection layer, visual backbone network and multi-head cross attention layer, outputs availability heatmap and grounding visual features; Policy Decoder: Employs a conditional denoising diffusion probability model to predict robot actions based on grounded visual features and proprioceptive state.

10. The availability-guided bidirectional latent planning system for long-range operation tasks according to claim 9, characterized in that, The system also includes a two-stage training module: first, a bidirectional latent planner is trained, and then the loss is predicted using a single action. End-to-end joint optimization of the availability grounding module and the policy decoder is performed to ensure the correlation between the visual characteristics of grounding and the final action generation.

Images