A language-conditioned visual history compression method and system

Abstract

The present application belongs to the technical field of robot intelligent control and embodied intelligence, and particularly relates to a language-conditioned visual history compression method, comprising the following steps: obtaining a historical observation sequence and a corresponding natural language task instruction; using a pre-trained visual encoder to perform feature coding on the historical observation sequence to obtain candidate historical feature tokens. The compression screening weight of the historical observation is dynamically regulated by the natural language task instruction, so that the retention of historical information is highly matched with the current task, fundamentally solving the defect of the "wrong information retention" of the task-independent compression mechanism; more task-critical visual information is retained under a fixed token budget, effectively improving the success rate of execution of long-range operation, occlusion scene, reference type instruction and other history-dependent tasks, and breaking through the Markov hypothesis limitation of the single-frame VLA model.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control and embodied intelligence technology, and particularly relates to a visual history compression method and system with language conditionalization. Background Technology

[0002] With the development of embodied intelligence technology, Vision-Language-Action (VLA) models have become the core control scheme for robot operation tasks, widely used in scenarios such as desktop assembly, item sorting, and human-robot collaboration. In actual long-term operation tasks, the robot's current decision-making generally relies on historical observation information, such as the position of occluded target objects, completed process steps, past object placement states, and referential descriptions of past events in natural language instructions. Such tasks do not satisfy the Markov assumption, and VLA models that only use the current single-frame visual observation as input have inherent structural limitations and cannot effectively complete long-term operation tasks with strong historical dependencies.

[0003] To incorporate historical observation information into the VLA model, existing technologies primarily employ the method of directly stacking multiple historical observation frames, that is, inputting consecutive historical images along with the current frame into the vision-language model for processing. This approach has significant drawbacks in practical applications: stacking multiple frames greatly increases the total number of visual tokens, consumes the model's attention budget, dilutes the attention weights of the current frame and key information, and even introduces a large amount of task-irrelevant historical noise, causing the model to overfit to irrelevant cues and thus reducing task performance. Furthermore, in real-time robot control scenarios, constraints on computing power and inference latency make it difficult to implement the input of all historical frames, failing to support the incorporation of historical information over long windows.

[0004] Another existing approach uses historical observation compression. It encodes historical observations using a pre-trained visual encoder and then compresses all historical features into a fixed number of memory tokens through sparsification and summarization to control the total number of tokens. However, existing compression mechanisms are generally task-agnostic, with compression rules based on pre-defined, fixed logic, such as resampling based on learnable static query vectors or patch selection based on video encoding / decoding or motion statistics. The compression process is not guided by current task instructions. This characteristic leads to the tendency, under a fixed token budget, to retain visually significant but irrelevant information while omitting crucial but visually gradual content that is highly relevant to the task, resulting in the problem of "retaining incorrect information." Furthermore, some historical processing solutions rely on robot ontological perception information to assist feature extraction, requiring the simultaneous collection of ontological data such as joints and poses. This makes cross-hardware platform migration difficult and prevents the direct expansion of training data using unannotated human demonstration videos.

[0005] Furthermore, current robot memory solutions primarily focus on high-level symbolic and task planning, constructing semantic memory bases, knowledge graphs, or symbolic task state machines through large language models to achieve task decomposition, progress tracking, and long-term planning. The linguistic conditionalization of these solutions operates on high-level reasoning and planning stages, with symbolic semantic information serving as the memory carrier. This approach does not penetrate the lower-level visual perception layer and cannot conditionally control the token-level compression process of original visual history observations. In other words, current technologies lack a solution that can dynamically determine which local features in visual history observations should be retained and which should be discarded through natural language task instructions. This gap limits the performance ceiling of VLA models in long-term complex tasks.

[0006] In summary, existing methods for introducing historical information into VLA models suffer from problems such as the compression process being disconnected from the task, insufficient accuracy in information retention, and limited deployment adaptability. There is an urgent need for a historical observation compression scheme that can dynamically filter historical visual information in conjunction with task instructions and is easy to integrate and deploy. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned technical problems by providing a language-conditional visual history compression method.

[0008] In view of this, the present invention provides a visual history compression method with language conditionalization, comprising: Obtain historical observation sequences and corresponding natural language task instructions; A pre-trained visual encoder is used to encode the historical observation sequence to obtain candidate historical feature tokens. The candidate historical feature tokens are compressed according to the natural language task instructions to generate a preset number of memory tokens. The information filtering weight of the compression process is dynamically determined by the natural language task instructions. The memory token is injected into the action generation path of the vision-language-action model, and the robot control action is output using the memory token as a supplementary condition.

[0009] Furthermore, the pre-trained visual encoder is a frozen pre-trained visual encoder; the encoded candidate historical feature tokens are processed by a trainable linear projection layer and a normalization layer, and then aligned to the hidden dimension of the visual-language model backbone. During the model training phase, the network parameters of the pre-trained visual encoder and the visual-language model backbone are frozen, and parameter training is performed only on the functional modules, projection layers, and action heads corresponding to the compression processing.

[0010] Furthermore, the compression processing of the candidate historical feature token according to the natural language task instructions specifically includes: Using the learnable query vector as the query end and the token embedding sequence corresponding to the natural language task instruction as the key end, multi-head attention operation is performed to derive the language-conditional query token. The candidate historical feature tokens are resampled using a multi-layer cross-attention and feedforward network based on the language-conditional query tokens, and then distilled into a preset number of memory tokens.

[0011] Furthermore, the memory token is injected into the action generation path using a late fusion approach: The memory token is concatenated along the sequence dimension into the conditional sequence output by the visual-language model backbone. The action head performs cross-attention processing on the concatenated sequence to generate robot control actions. The memory token is not fed back into the visual-language model backbone network.

[0012] Furthermore, the memory token is injected into the action generation path using an early fusion approach: A preset number of placeholder tokens are appended to the input embedding sequence of the vision-language model. The memory tokens replace the embedding vectors of the corresponding placeholder tokens. The vision-language model backbone performs unified feature processing on the input sequence containing the memory tokens and outputs it to the motion head to generate robot control actions.

[0013] Furthermore, the historical observation sequence is taken from a fixed third-person perspective camera; the current frame observation data is taken from multiple perspective cameras, including hand-eye perspective, and the current frame observation data is input into the visual-language model backbone network for feature processing. All input signals for history compression and language conditionalization processing contain only natural language task instructions and image / video data, without using robot body perception information; The candidate historical feature tokens are obtained through patch sparse selection processing based on video codec alignment.

[0014] A language-conditional visual history compression system, used to perform the aforementioned language-conditional visual history compression method, includes: The data acquisition module is used to acquire historical observation sequences and corresponding natural language task instructions; The historical encoding module is used to encode the historical observation sequence using a pre-trained visual encoder to obtain candidate historical feature tokens. The language condition compression module is used to perform compression processing on the candidate historical feature tokens according to the natural language task instructions, and generate a preset number of memory tokens. The information filtering weight of the compression processing is dynamically determined by the natural language task instructions. The injection and action generation module is used to inject the memory token into the action generation path of the vision-language-action model, and output robot control actions using the memory token as a supplementary condition.

[0015] Furthermore, the historical encoding module includes a pre-trained visual encoding unit in a frozen state, and a trainable projection and normalization unit. The pre-trained visual encoding unit is used to encode the historical observation sequence, and the projection and normalization unit is used to align the encoded candidate historical feature tokens to the hidden dimensions of the visual-language model backbone. During the model training phase, the network parameters of the pre-trained visual encoding unit and the visual-language model backbone are frozen, and only the language conditional compression module, projection and normalization unit, and action head participate in parameter training.

[0016] Furthermore, the language conditional compression module includes a query generation unit and a resampling unit; The query generation unit is used to perform multi-head attention operation with a learnable query vector as the query end and the token embedding sequence corresponding to the natural language task instruction as the key end, to derive a language-conditional query token; The resampling unit is used to perform multi-layer cross-attention and feedforward network resampling processing on candidate historical feature tokens using the language-conditional query token, and distill to obtain a preset number of memory tokens.

[0017] Furthermore, the injection and action generation module includes a late fusion unit and an early fusion unit; The late fusion unit is used to concatenate the memory token to the conditional sequence output by the visual-language model backbone and send it to the action head, and the memory token does not flow back to the visual-language model backbone network; The early fusion unit is used to replace the placeholder token in the input sequence of the visual-language model with the memory token, which is then processed uniformly by the visual-language model backbone and sent to the action head.

[0018] The beneficial effects of this invention are: By dynamically adjusting the compression and filtering weights of historical observations according to natural language task instructions, the retention of historical information is highly matched with the current task, fundamentally solving the defect of "retaining wrong information" in task-irrelevant compression mechanisms. Under a fixed token budget, more key visual information of the task is retained, effectively improving the execution success rate of historically dependent tasks such as long-range operations, occluded scenes, and referential instructions, and breaking through the Markov assumption limitation of single-frame VLA models. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture and data flow of the visual history compression method for language conditionalization of the present invention; Figure 2 This is a schematic diagram of the internal structure of the language conditional compression module of the present invention; Figure 3 This is a comparative diagram of two methods for injecting the memory token in this invention; Figure 4 This is a schematic diagram illustrating the input configuration of the current frame from multiple perspectives and the history from a single third-person perspective in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] In the description of this application, it should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. For ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0022] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] It should be noted that in the description of this application, the directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0024] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0025] Reference Figures 1 to 4 A visual history compression method based on language conditionation includes the following steps: A historical observation sequence with a preset window length and its corresponding natural language task instructions are obtained. A pre-trained visual encoder is used to perform frame-by-frame feature encoding on the historical observation sequence, and a candidate historical feature token set is obtained after sparsification filtering. The candidate historical feature token set is compressed according to the natural language task instructions to generate a preset fixed number of memory tokens. The information filtering weight of the compression processing is dynamically determined by the natural language task instructions. The memory tokens are injected into the action generation path of the vision-language-action model, and the robot control actions are output using the memory tokens as historical supplementary conditions.

[0026] By dynamically adjusting the compression and filtering weights of historical observations according to natural language task instructions, the retention of historical information is highly matched with the current task, fundamentally solving the defect of "retaining wrong information" in task-irrelevant compression mechanisms. Under a fixed token budget, more key visual information of the task is retained, effectively improving the execution success rate of historically dependent tasks such as long-range operations, occluded scenes, and referential instructions, and breaking through the Markov assumption limitation of single-frame VLA models.

[0027] In this embodiment, the pre-trained visual encoder is a frozen ViT visual encoder aligned with HEVC encoding and decoding. The pre-trained weights are trained based on a large-scale image-video dataset. The single-frame encoded output is a patch-level feature sequence with an example feature dimension of 768 dimensions. The encoded candidate historical feature tokens are sequentially passed through a trainable linear projection layer and a LayerNorm normalization layer to accurately align the feature dimensions to the hidden dimensions of the visual-language model backbone.

[0028] During the model training phase, all network parameters of the pre-trained visual encoder and all network parameters of the visual-language model backbone are frozen, and parameter updates and training are performed only on the functional modules, linear projection layers, normalization layers, and action head networks corresponding to the compression processing.

[0029] By adopting the frozen pre-trained encoder approach, the visual encoding can always maintain the general representation ability under the pre-trained distribution, avoiding problems such as encoder representation degradation and overfitting under small sample tasks. Only lightweight compression modules, projection layers and action heads are trained, resulting in a small number of training parameters, fast convergence speed and high training stability. It can be plugged and played into existing mature VLA models without the need to reconstruct the backbone network, which greatly reduces the development and training costs of the solution.

[0030] In this embodiment, the compression processing of the candidate historical feature token according to the natural language task instructions is implemented using a two-level attention architecture, specifically including: The first step, query derivation: initialize N learnable query vectors (N is the preset number of memory tokens, the example value is 8), use the learnable query vector as the query end Q, use the token embedding sequence output by the language encoding layer of the natural language task instruction as the key end K and the value end V, perform 8-head multi-head attention operation, and output N language-conditional query tokens; this process injects task semantics into the query vector, so that the query target is accurately aligned with the current task instruction.

[0031] The second step is resampling: using the language-conditionalized query token as the query end and the candidate historical feature token as the key and value ends, a stacked structure of two layers of "cross attention + feedforward network (FFN)" is executed. Each layer of cross attention is followed by residual connection and layer normalization. Finally, the K candidate historical feature tokens (example value 512) are distilled into N fixed number of memory tokens.

[0032] By employing a two-level attention mechanism of "language-derived query vector + cross-attention resampling", natural language semantics are injected into the visual history compression process, truly achieving task-adaptive compression where "the same historical observation outputs different memory tokens for different task instructions". This resampling structure is fully differentiable, forming a learnable information bottleneck that can filter out noise interference and attention dilution from task-irrelevant visual patches, while also supporting end-to-end joint optimization to ensure that the compressed memory token is highly relevant to the task objective.

[0033] As an equivalent alternative implementation, the compression process performed according to the natural language task instructions can also be implemented in any of the following ways: The first method involves deriving a relevance vector from a natural language task instruction through encoding, calculating the dot product relevance score between each candidate historical feature token and the relevance vector, additively fusing the relevance score with the visual saliency score of the candidate historical feature token itself, performing soft selection or weighted sampling on the candidate historical feature token based on the fused score, and aggregating to obtain a preset number of memory tokens. The second method involves inputting the embedding vector of the natural language task instruction into the multilayer perceptron (MLP) scoring network, outputting the relevance weight corresponding to each candidate historical feature token, and then weighting and aggregating all candidate historical feature tokens based on the relevance weight to obtain the memory token. The third method involves calculating the relevance scores between each candidate historical feature token and the natural language task instructions, sorting them in descending order of score, and selecting the top-K candidate historical feature tokens as memory tokens to achieve hard-screening and compression.

[0034] It provides multiple equivalent language conditional compression implementation paths, which can be flexibly selected according to computing power constraints, deployment platforms and task characteristics; among them, soft selection and weighted aggregation methods have good differentiability and are suitable for end-to-end training; hard Top-K selection has the lowest computational cost and is suitable for edge deployment scenarios with extremely low computing power. The overall adaptability and deployment flexibility of the solution are significantly improved.

[0035] In this embodiment, the memory token injection action generation path adopts a late fusion approach: N memory tokens are concatenated along the sequence dimension to the end of the final hidden state sequence output by the visual-language model backbone, forming a complete conditional sequence containing the current frame semantics, language instructions, and historical memories. The flow-matching action head takes this conditional sequence as input, performs cross-attention feature aggregation, and then generates a single-step robot control action through a flow-matching diffusion process. An example of this action dimension is 7-dimensional, including 6-dimensional end-effector pose and 1-dimensional gripper opening. In this method, the memory tokens are only input to the action head and do not flow back to the visual-language model backbone network.

[0036] In the late fusion approach, the memory token does not enter the VLM backbone, thus not increasing the computational load and inference latency of the backbone network, making it suitable for low-computing-power edge deployment scenarios. At the same time, this injection method is highly compatible with the action heads of existing mainstream VLA architectures, allowing for rapid integration without modifying the backbone network structure, which facilitates comparison and verification and scheme migration on different baseline models.

[0037] In this embodiment, the memory token injection action generation path adopts an early fusion approach: In the input embedding sequence of the vision-language model, N placeholder tokens are appended between the current frame visual token and the language instruction token. The embedding vectors of the corresponding placeholder tokens are replaced by N memory tokens to form an input sequence of "current frame visual token + memory token + language instruction token". This input sequence is fed into the backbone of the vision-language model, and through multi-layer self-attention operations, the deep fusion of historical information, current observation and language instructions is achieved. The final hidden state sequence output by the backbone is fed into the flow matching action head to generate robot control actions.

[0038] In the early fusion approach, memory tokens can participate in multi-layer attention calculations of the VLM backbone, enabling deep multi-round feature fusion of historical information, current frame observations, and language instructions, resulting in more complete semantic alignment. For complex long-range tasks and strong referential instructions, it can further improve the accuracy of action decisions, making it suitable for application scenarios with sufficient computing power and performance priority.

[0039] In this embodiment, the input data and viewpoint are configured as follows: The historical observation sequence is taken from a third-person global camera that is fixedly installed in the robot's working scene. The frame rate is matched with the robot's control frequency. The historical window length T is 32 frames in the example, covering the operation process of the past few seconds. The current frame observation data is taken from a multi-view camera including hand-eye-wrist cameras and third-person cameras. The images from each view are encoded separately and then spliced ​​along the sequence dimension, and then input into the visual-language model backbone network.

[0040] The entire input for history compression and language conditionalization consists only of natural language task instructions and image / video data, without accessing proprioceptive signals such as robot joint angles and end-effector poses.

[0041] The candidate historical feature tokens are obtained through patch sparse selection based on HEVC encoding / decoding alignment: using the motion vectors output by the HEVC encoding process, the motion amplitude of each image patch is calculated, and the top 30% of patches in each frame are selected. After cross-frame deduplication, these patches are uniformly sampled to a fixed number K (Example 512) to form a set of candidate historical feature tokens. As an equivalent alternative, the sparse selection can also be implemented using motion saliency filtering based on optical flow, change region filtering based on inter-frame difference, or patch filtering based on a learned motion saliency network. All of these methods can achieve pre-filtering of redundant and invariant patches.

[0042] All input images were uniformly scaled to 224×224 resolution during the preprocessing stage, and normalized using the mean and standard deviation of the ImageNet dataset.

[0043] It adopts a fixed third-person perspective to record historical observations, with stable camera position and full field of view covering the entire working area. It can continuously record changes in object position, occlusion relationships and task progress, avoiding fragmentation of historical information caused by hand-eye camera shake and narrow field of view, and is more suitable for long-term memory storage; it only encodes and compresses single-channel historical video, significantly reducing computing power and token overhead.

[0044] It does not rely on ontology perception information, does not require synchronous collection of ontology data such as robot joints and poses, has low migration costs across hardware platforms, and can directly utilize human operation demonstration videos without ontology annotations to expand the training dataset.

[0045] Multiple pre-process sparse filtering methods can filter out a large number of unchanged redundant patches in advance, significantly reduce the size of candidate historical tokens, further reduce the computational load of subsequent compression modules, and improve the real-time performance of inference.

[0046] In this embodiment, the entire model training process is configured as follows: An end-to-end supervised training approach is adopted. The sample construction adopts a sliding window method to extract training samples from the demonstration video. Each sample contains T frames of historical observation, 1 frame of current multi-view observation, 1 natural language command and a corresponding 1 frame of action truth label. The sliding step size is 8 frames. The training data can simultaneously use robot-autonomous data collection data and human operation demonstration videos without ontology annotation.

[0047] The loss function consists of two parts: the main loss is the training loss of the action head: when the action head adopts a flow-matching structure, it is the velocity field regression loss, which calculates the mean square error between the predicted velocity field and the target velocity field; when the action head adopts a regression structure, it is the L1 or mean square error (MSE) loss between the predicted action and the ground truth action; an auxiliary loss for subtask classification is added, in which the memory token is global average pooled and then fed into a 2-layer MLP classifier to output the subtask category prediction. The cross-entropy loss is used to calculate the subtask classification error, and the auxiliary loss weight is set to 0.1 to enhance the task information carrying capacity of the memory token.

[0048] The optimized configuration uses the AdamW optimizer, with an initial learning rate of 3e-4, weight decay of 1e-4, and a training batch size of 32. The cosine annealing learning rate scheduling strategy is adopted, and the number of training epochs is dynamically adjusted according to the dataset size.

[0049] Using a sliding window to construct samples can make full use of long-term time-series demonstration data and improve the task coverage of historical windows; the introduction of auxiliary loss can explicitly constrain the task relevance of memory tokens, avoid degradation in the compression process, and further improve the compression quality; the training strategy of freezing the backbone, combined with the standard optimizer configuration, ensures stable convergence of the training process, has strong reproducibility, and can be quickly implemented and verified on existing VLA bases.

[0050] In addition to the aforementioned short-term visual history compression path, a symbolic long-term memory branch can be maintained in parallel, as specifically implemented below: The complete task is pre-decomposed into an ordered sequence of atomic subtasks, and the sequence of completed subtasks is maintained as long-term memory data. A subtask predictor is set up, which adopts a 2-4 layer Transformer structure with learnable query vectors. Based on the current visual-language understanding results and the sequence of completed subtasks, the latent variable z of the current task is predicted in the language embedding space. The current task latent variable z has two functions: first, it is input into the query generation unit to participate in the derivation process of the language conditional query token, further strengthening the guidance of long-term task progress on historical compression; second, it is concatenated along the sequence dimension into the conditional sequence of the action generation path to assist the action head in long-term task progress perception and action decision-making.

[0051] By combining short-term visual memory with long-term task progress memory through parallel symbolic long-term memory branches, it can support complex long-term tasks with more steps and stronger temporal dependencies. The symbolic memory at the subtask level and the compressed memory at the visual level complement each other, which not only retains fine-grained visual information, but also has the ability to control the progress of high-level tasks, further expanding the task coverage boundary of the solution.

[0052] A language-conditional visual history compression system, used to perform the language-conditional visual history compression method described above, comprising: The data acquisition module is used to acquire historical observation sequences of a preset window length and the corresponding natural language task instructions; The historical encoding module is used to perform frame-by-frame feature encoding on the historical observation sequence using a pre-trained visual encoder, and obtain a set of candidate historical feature tokens after sparsification filtering. The language condition compression module is used to perform compression processing on the candidate historical feature token set according to the natural language task instruction, and generate a preset fixed number of memory tokens. The information filtering weight of the compression processing is dynamically determined by the natural language task instruction. The injection and action generation module is used to inject the memory token into the action generation path of the vision-language-action model, and output robot control actions using the memory token as a historical supplementary condition.

[0053] Adopting a modular architecture design, each module has clear functional boundaries and well-defined input and output. It can be embedded as an independent perception and memory unit into the existing vision-language-motion control system. It has the same technical performance as the corresponding method, and can realize task-adaptive compression of historical visual information to improve the robot's ability to perform long-range operation tasks.

[0054] In this embodiment, the historical encoding module includes a pre-trained visual encoding unit in a frozen state, a sparse filtering unit, and a trainable projection and normalization unit. The pre-trained visual encoding unit adopts the ViT architecture aligned with HEVC encoding and decoding, and is used to encode features frame by frame in the historical observation sequence; the sparse filtering unit is used to sparsely filter the encoded patch features based on motion amplitude to obtain a fixed number of candidate historical feature tokens; the projection and normalization unit is used to align the feature dimensions of the candidate historical feature tokens to the hidden dimensions of the visual-language model backbone.

[0055] During the model training phase, all network parameters of the pre-trained visual encoding unit and the visual-language model backbone are frozen, and only the language conditional compression module, projection and normalization unit, and action head participate in parameter training.

[0056] By freezing the parameter configuration of the coding unit and the backbone network, the stability of the system's visual representation capability is ensured; the sparse filtering unit is placed in front to reduce the computational load of subsequent modules; the lightweight projection and normalization units achieve smooth adaptation of feature dimensions; the overall training cost is low and the integration difficulty is small, and it can be quickly adapted to VLM backbone networks of different specifications.

[0057] In this embodiment, the language conditional compression module includes a query generation unit and a resampling unit; The query generation unit is used to perform multi-head attention operation with a learnable query vector as the query end and the token embedding sequence corresponding to the natural language task instruction as the key end to derive a language-conditional query token; The resampling unit is used to perform multi-layer cross-attention and feedforward network resampling processing on candidate historical feature tokens using the language-conditional query token, distilling a large number of candidate historical feature tokens into a preset fixed number of memory tokens.

[0058] The two-level structure of query generation unit and resampling unit enables precise control of visual compression process by language semantics; the unit adopts standard attention and feedforward network structure, which is highly interpretable and maturely optimized, ensuring the stability and task relevance of memory token generation.

[0059] In this embodiment, the injection and action generation module includes a late fusion unit, an early fusion unit, and a stream matching action head unit; The late fusion unit is used to concatenate the memory token to the conditional sequence output by the visual-language model backbone and send it to the stream matching action head unit, and the memory token does not flow back to the visual-language model backbone network; The early fusion unit is used to replace the placeholder token in the input sequence of the visual-language model with the memory token, and after unified processing by the visual-language model backbone, it is sent to the stream matching action head unit. The flow matching action head unit is used to generate robot control actions based on the input conditional sequence.

[0060] It is equipped with two fusion units, which can flexibly switch the injection mode according to the computing power constraints and performance requirements of the deployment scenario; it is compatible with motion heads of different architectures, adapts to diverse robot control hardware and task scenarios, and improves the versatility of the system.

[0061] In an optional extended embodiment, the system further includes a long-term memory module; The long-term memory module is used to maintain the sequence of completed subtasks as long-term memory data, and the subtask predictor predicts the latent variables of the current task based on the current semantic understanding results and the sequence of completed subtasks. The current task latent variables are input to the language condition compression module to participate in query derivation, and / or input to the injection and action generation module to participate in action decision condition concatenation.

[0062] Modular long-term memory branches can be plugged in and out as needed to adapt to task scenarios of varying complexity; working in conjunction with short-term visual compression modules, they form a hierarchical memory system that combines long and short-term memory, enhancing the system's adaptability to complex long-term tasks.

[0063] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A visual history compression method with linguistic conditionalization, characterized in that, Includes the following steps: Obtain historical observation sequences and corresponding natural language task instructions; A pre-trained visual encoder is used to encode the features of the historical observation sequence to obtain candidate historical feature tokens; The candidate historical feature tokens are compressed according to the natural language task instructions to generate a preset number of memory tokens. The information filtering weight of the compression process is dynamically determined by the natural language task instructions. The memory token is injected into the action generation path of the vision-language-action model, and the robot control action is output using the memory token as a supplementary condition. The pre-trained visual encoder is a pre-trained visual encoder in a frozen state. The encoded candidate historical feature tokens are processed by a trainable linear projection layer and a normalization layer, and then aligned to the hidden dimensions of the visual-language model backbone. During the model training phase, the network parameters of the pre-trained visual encoder and the visual-language model backbone are frozen, and parameter training is performed only on the functional modules, projection layers, and action heads corresponding to the compression processing.

2. The visual history compression method with language conditionalization according to claim 1, characterized in that, The compression process performed on the candidate historical feature token according to the natural language task instructions specifically includes: Using the learnable query vector as the query end and the token embedding sequence corresponding to the natural language task instruction as the key end, multi-head attention operation is performed to derive the language-conditional query token. The candidate historical feature tokens are resampled using a multi-layer cross-attention and feedforward network based on the language-conditional query tokens, and then distilled into a preset number of memory tokens.

3. The visual history compression method with language conditionalization according to claim 1, characterized in that, The compression process performed on the candidate historical feature token according to the natural language task instructions specifically includes: A relevance feature vector is derived from the natural language task instructions. The relevance weight between each candidate historical feature token and the relevance feature vector is calculated. Based on the relevance weight, all candidate historical feature tokens are weighted and aggregated to obtain a preset number of memory tokens.

4. The visual history compression method with language conditionalization according to claim 1, characterized in that, The memory token is injected into the action generation path using a late fusion method: The memory token is concatenated along the sequence dimension into the conditional sequence output by the visual-language model backbone. The action head performs cross-attention processing on the concatenated sequence to generate robot control actions. The memory token is not fed back into the visual-language model backbone network. The memory token is injected into the action generation path using an early fusion approach: A preset number of placeholder tokens are appended to the input embedding sequence of the visual-language model. The memory tokens replace the embedding vectors of the corresponding placeholder tokens. The visual-language model backbone performs unified feature processing on the input sequence containing the memory tokens and outputs it to the motion head to generate robot control actions. The historical observation sequence is taken from a fixed third-person perspective camera; the current frame observation data is taken from multiple perspective cameras including the hand-eye perspective, and the current frame observation data is input into the visual-language model backbone network for feature processing. All input signals for history compression and language conditionalization processing contain only natural language task instructions and image / video data, without using robot body perception information; The candidate historical feature tokens are obtained through patch sparsification filtering based on motion features.

5. The visual history compression method with language conditionalization according to claim 4, characterized in that, The action head is a flow-matching action head, trained using velocity field regression loss; or the action head is a regression-based action head, trained using L1 loss or mean squared error loss.

6. A visual history compression system with language conditionation, characterized in that, A visual history compression method for performing language conditionalization as described in any one of claims 1-5, comprising: The data acquisition module is used to acquire historical observation sequences and corresponding natural language task instructions; The historical encoding module is used to encode the historical observation sequence using a pre-trained visual encoder to obtain candidate historical feature tokens. The language condition compression module is used to perform compression processing on the candidate historical feature tokens according to the natural language task instructions, and generate a preset number of memory tokens. The information filtering weight of the compression processing is dynamically determined by the natural language task instructions. The injection and action generation module is used to inject the memory token into the action generation path of the vision-language-action model, and output robot control actions using the memory token as a supplementary condition.

7. The language-conditional visual history compression system according to claim 6, characterized in that, The historical encoding module includes a pre-trained visual encoding unit in a frozen state, as well as a trainable projection and normalization unit. The pre-trained visual encoding unit is used to encode the historical observation sequence, and the projection and normalization unit is used to align the candidate historical feature tokens to the hidden dimensions of the visual-language model backbone. During the model training phase, the network parameters of the pre-trained visual encoding unit and the visual-language model backbone are frozen, and only the language conditional compression module, projection and normalization unit, and action head participate in parameter training.

8. The visual history compression system with language conditionalization according to claim 7, characterized in that, The language conditional compression module includes a query generation unit and a resampling unit; The query generation unit is used to perform multi-head attention operation with a learnable query vector as the query end and the token embedding sequence corresponding to the natural language task instruction as the key value end, to derive a language-conditional query token; The resampling unit is used to perform multi-layer cross-attention and feedforward network resampling processing on candidate historical feature tokens through the language-conditional query token, and distill to obtain a preset number of memory tokens; The injection and action generation module includes a late fusion unit and an early fusion unit; The late fusion unit is used to concatenate the memory token to the conditional sequence output by the visual-language model backbone and send it to the action head, and the memory token does not flow back to the visual-language model backbone network; The early fusion unit is used to replace the placeholder token in the input sequence of the visual-language model with the memory token, which is then processed uniformly by the visual-language model backbone and sent to the action head.

9. The visual history compression system with language conditionalization according to claim 8, characterized in that, The action head used by the injection and action generation module is either a stream-matching action head or a regression action head trained with L1 loss or mean squared error loss.

Images