Robot control method based on hierarchical compact visual motion memory

Abstract

The application provides a robot control method based on hierarchical compact visual motion memory, and belongs to the technical field of robot learning and embodied intelligence. The current visual observation, task instruction and robot state are acquired, a visual token with space-time position coding is generated by using a visual encoder, and three-level memories of short-term memory, medium-term memory and long-term memory are constructed under a fixed token capacity. When action decoding is performed, an adaptive modulation mechanism is used to retrieve historical information from the three-level memories, and the retrieval result is converted into a scale parameter and an offset parameter to modulate the hidden state of an action expert, thereby guiding the generation of an action sequence. The application can reduce the redundancy and computational overhead of visual historical information, avoid the loss of key information, and improve the execution performance of the robot in operation tasks with long-term dependence and state confusion.

Description

Technical Field

[0001] This invention relates to the field of robot learning and embodied intelligence technology, and in particular to a robot control method based on hierarchical compact visual-motor memory. Background Technology

[0002] Vision-language-action models can achieve end-to-end control from environmental perception to robot action generation by jointly modeling visual observations, natural language task instructions, and robot state information, demonstrating good generalization ability in complex robot manipulation tasks. However, most existing robot manipulation methods predict actions based on current visual observations, implicitly assuming that the current observation information can completely describe the robot's state, i.e., satisfying the Markov property. In real physical environments, robots often face situations such as visual occlusion, changes in object state, long-term task dependence, and the inability to directly observe historical interaction information. Relying solely on current observations can easily lead to state confusion, making it impossible for the robot to accurately judge task progress and historical operation results, thereby reducing the success rate of non-Markov tasks.

[0003] To address the aforementioned issues, existing methods attempt to enhance robot maneuverability by incorporating historical memory. Dense history caching methods, by storing a large amount of historical observations or visual features, enable the robot to acquire complete historical information, but this incurs significant storage overhead and visual redundancy, reducing inference efficiency. Another type of method uses keyframe selection or sparse sampling strategies to reduce computational costs, but inevitably loses continuous visual motion interaction information, such as object trajectories and changes in contact states. Furthermore, some memory enhancement methods directly concatenate historical information into the model input; as the length of the history increases, computational complexity rises, and irrelevant information is introduced, affecting the stability of action prediction. Therefore, existing technologies struggle to simultaneously meet the requirements of historical information integrity, long-term reliance on modeling capabilities, and real-time inference efficiency. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a robot control method based on hierarchical compact visual motion memory. By applying different compression intensities according to the time distance between historical observations and the current moment, and combining a spatiotemporal dynamic selection module, an instruction guidance aggregation module, and an adaptive memory modulation module, this method retains historical visual information related to robot operations within a fixed memory token capacity. This invention can reduce the redundancy of visual historical information and computational overhead while avoiding the loss of key information and improving the robot's performance in non-Markovian operation tasks.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a robot manipulation method based on hierarchical compact visual motion memory, comprising the following steps: S1. Obtain visual motion history information: at time step Obtain the robot's current visual observation Task instructions and robot status information and utilize visual encoders Current visual observation Image mapped to a set of visual tokens , where each visual token All include time location codes and spatial location coding ; S2. Constructing hierarchical visual motion memory: During continuous robot operation, the historical observation sequence is... Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints It is also divided into short-term memory based on time scale. Mid-term memory and long-term memory ; S3, Progressive Visual-Motor Memory Compression: This method compresses data written into short-term memory. Visual tokens Perform parameterless average pooling to generate short-term visual tokens; when short-term memory... Not exceeding its capacity At that time, short-term visual tokens are written into short-term memory in chronological order. When short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory And filter out spatiotemporal redundant tokens; during the middle period memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. ; S4, Visual-Motor Memory Injection: The short-term, medium-term, and long-term memories processed in S3 are concatenated and converted into a memory token sequence, which is then injected into the motion expert. Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Perform adaptive modulation; S5. Output result: Hides state using modulated motion. Combined with current visual observations Task instructions and robot status information The initial noisy action sequence is iteratively decoded to output the action sequence used to control the robot. After the action is executed, the new observation is added to the historical observation sequence to achieve continuous memory updates.

[0006] The beneficial effects of this invention are as follows: This invention solves the problems of insufficient utilization of historical information and low memory efficiency in existing robot control methods by using hierarchical compact visual motion memory. It progressively compresses historical visual motion information through a three-level memory structure of short-term memory, medium-term memory, and long-term memory. This reduces visual information redundancy and computational overhead while retaining key visual cues relevant to the task and avoiding the loss of continuous visual motion interaction information. At the same time, through an adaptive memory modulation mechanism, historical information is effectively injected into the action generation process, improving the accuracy and stability of action generation and execution in non-Markovian operation tasks such as long-term dependence, visual occlusion, and state confusion.

[0007] Furthermore, the construction of hierarchical visual-motor memory includes the following steps: From the initial time step to the current time step The historical observation sequence is denoted as Among them, current visual observation After visual encoder Visual tokens obtained from mapping Visual historical information, as the current time step, is written into the historical observation sequence; During the continuous operation of the robot, the historical observation sequence will be used. Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints :

[0008] in, Indicates by task instructions The memory compression map consists of write, select, aggregate, and update operations. Indicates the total visual token capacity. This represents the feature dimension of the unified memory token; Representing compact visual memory Based on time scale, it is divided into short-term memory. Mid-term memory and long-term memory ,satisfy:

[0009]

[0010] in, Indicates time step A hierarchical and compact set of visual memory, Represents hierarchical index The corresponding subset of memories, Represents hierarchical index The corresponding capacity limit, Indicates short-term memory levels. Indicates the intermediate memory level. Indicates the long-term memory hierarchy. , and They represent short-term memory. Maximum capacity, intermediate memory Maximum capacity and long-term memory Maximum capacity, This indicates the total visual token capacity.

[0011] The beneficial effects of the above-mentioned further solutions are: the most recent frames need to be densely retained to achieve basic localization, while the distant historical records can tolerate a stronger degree of compression. This invention establishes a three-level memory structure of short-term memory, medium-term memory, and long-term memory under a fixed total visual token capacity, allowing historical visual information at different time scales to be stored with different compression intensities. This avoids the continuous accumulation of complete historical visual tokens, ensuring that the amount of historical feature cache and additional computation does not increase linearly with the robot's operation time.

[0012] Furthermore, the writing to short-term memory Visual tokens Perform parameterless average pooling to generate short-lived visual tokens, including the following steps: For writing to short-term memory The Frame visual token Divided into Non-overlapping local spatial regions And perform average pooling within each local spatial region to generate short-lived visual tokens:

[0013] in, This represents the first value obtained after average pooling. A short-term visual token Indicates the first A local spatial region, This indicates the number of visual tokens contained in the area. Indicates the first Frame space location Visual tokens at the location; The expression for the spatiotemporal location encoding corresponding to the short-term visual token is as follows:

[0014] in, Indicates the first Frame number Each pooled region corresponds to the spatiotemporal location encoding of the visual token. Indicates the first The location of each pooled region within the compressed spatial grid.

[0015] The beneficial effects of the above-mentioned further scheme are: parameterless average pooling does not require the introduction of additional learnable parameters, and can retain the overall semantics and spatial layout of recent visual observations while reducing the number of visual tokens in a single frame, providing more complete global perception information for current action decisions.

[0016] Furthermore, the aforementioned short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory This includes the following steps: When short-term memory Exceeding its capacity At that time, for the current overflow frame and its adjacent reference frame Calculate visual tokens for the same spatial region Time series change score:

[0017] in, Indicates the first The temporal variation score of the visual token corresponding to each local spatial region. Indicates the current overflow frame The Middle Short-term visual tokens corresponding to local spatial regions Indicates adjacent reference frames The Middle Short-term visual tokens corresponding to local spatial regions This represents a preset constant to prevent the denominator from being zero; The local spatial region with the highest temporal change score is identified as the current overflow frame. Time change anchor point :

[0018] in, This operation represents the operation of returning the index of the region corresponding to the maximum temporal change score from all local spatial regions; Time-varying anchor points Calculate the local spatial region Chebyshev distance between the corresponding visual token and the time-varying anchor point:

[0019] in, Represents a local spatial region Anchor points of time change Chebyshev distance between them Represents a local spatial region Row coordinates in a compressed space grid Anchor points representing time changes Row coordinates in a compressed space grid Represents a local spatial region Column coordinates in a compressed space grid. Anchor points representing time changes Column coordinates in a compressed spatial grid; Domain-based reweighting of temporal variation scores based on Chebyshev distance:

[0020] in, Represents the i-th element after spatial neighborhood reweighting. The spatiotemporal saliency score of a local spatial region Anchor points representing time changes The radius of the surrounding spatial neighborhood, This represents the enhancement coefficient of the neighborhood region. This represents the scaling factor for the area outside the neighborhood; Based on the temporal change scores after domain reweighting, the top M key visual tokens are selected. Write into intermediate memory :

[0021]

[0022] in, Indicates the current overflow frame The mid-term visual token set obtained after spatiotemporal dynamic selection Key visual token index, Indicates an overflow frame Write into intermediate memory Key visual tokens Indexed collection, Represents the real number field. Represents the dimension of visual features.

[0023] The beneficial effects of the above-mentioned further scheme are: by comparing visual tokens at the same spatial position in adjacent historical frames, it is possible to identify visual regions that have undergone significant semantic changes, retain dynamic information such as object movement, robotic arm interaction and object state changes, while filtering out static backgrounds that recur in adjacent frames. By introducing a spatial neighborhood bias centered on the time-varying anchor point, the spatiotemporal dynamic selection not only retains visual tokens with high change scores, but also preserves the local context around the significantly changed areas, avoiding the selected visual tokens being too spatially discrete and improving the ability of mid-term memory to express continuous visual motion processes.

[0024] Furthermore, the aforementioned intermediate memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. This includes the following steps: When intermediate memory Exceeding its capacity At that time, for the frame to be migrated Based on the highest M key visual tokens Calculate each candidate visual token retained by spatiotemporal dynamic selection. With task instructions The correlation score between the text and images:

[0025] in, Represents intermediate memory Candidate visual tokens With task instructions The image-text relevance score between query vectors. and These represent the visual projection matrix and the speech projection matrix, respectively. Represents the set of candidate visual tokens. Represents a text query vector. Indicates the candidate visual token index. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; The image-text relevance score is fused with the spatiotemporal saliency score obtained during the spatiotemporal dynamic selection phase to obtain a comprehensive selection score, and the L candidate visual tokens with the highest comprehensive scores are selected:

[0026] in, This represents the set of visual token indices that participate in long-term aggregation. and These represent the image-text relevance scores, respectively. Spatiotemporal saliency score The control coefficient, This indicates that the candidate visual tokens with the highest overall score are selected. Operations corresponding to the index of each token. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; Normalization with a temperature parameter is performed on the L candidate visual tokens to obtain the aggregate weights:

[0027] in, Indicates the first The aggregate weight of each candidate visual token. Indicates temperature parameter, Represents the visual token index set Elements in; The candidate visual tokens are weighted and summed according to the aggregation weights to generate perceptual anchors for long-term memory:

[0028] in, This represents the perceptual anchor point that is written into long-term memory. Represents the real number field. Indicates the dimension of visual features; Write the perceptual anchors of long-term memory into long-term memory .

[0029] The beneficial effect of the above-mentioned further scheme is that by further introducing task instruction relevance into the intermediate candidate visual tokens, long-term memory pays more attention to historical visual information related to the current manipulation target, target object and operation semantics, and reduces the interference of task-irrelevant dynamic regions on long-term action decision-making. By comprehensively considering the relevance of task instructions and the spatiotemporal dynamic saliency, and aggregating multiple candidate visual tokens into a single perceptual anchor, it is possible to retain key visual semantics relevant to the current task while significantly compressing long-term historical information, thus forming a compact long-term memory representation with task-discriminative capabilities.

[0030] Furthermore, step S4 includes the following steps: The short-term, medium-term, and long-term memories processed by S3 are concatenated and mapped into a hierarchical memory sequence. Action expert Hidden state of layer actions As a query, hierarchical memory sequence Perform cross-attention as keys and values ​​to obtain a memorized conditional representation:

[0031] in, For the purpose of memory condition representation, , and These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. Represents the attention feature dimension. This represents the memory mask generated based on the valid memory token; Representing memory conditions Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Perform adaptive modulation:

[0032]

[0033] in, This indicates the hidden state of actions modulated by memory. This indicates the hidden state of the next level of the motion expert. Indicates the gating coefficient. This indicates the current denoising time step for the motion expert.

[0034] The beneficial effects of the above-mentioned further scheme are: by actively querying the hierarchical memory through the action hidden state, the action expert can selectively retrieve historical information based on the current action generation state, avoiding the direct splicing of the complete historical visual sequence into the input context of the visual language model, thereby avoiding the context length from increasing with the number of historical observations. By modulating the action hidden state at the feature level using scale and offset parameters, hierarchical historical memory can directly influence the action feature representation within the action expert, thereby improving the ability of long-term historical information to guide the action generation process without changing the original input structure of the visual language model.

[0035] The present invention has at least the following beneficial effects: 1) Based on the differences in the role of historical information at different time scales, hierarchical storage is performed so that recent visual observations retain complete details, mid-term visual observations retain dynamic interactions, and long-term visual observations retain task-related semantics, thereby improving the information fidelity in non-Markov tasks. 2) The overall semantics of recent visual observations are preserved by parametric average pooling, and key visual tokens with significant semantic changes and spatial neighborhood consistency are preserved by spatiotemporal dynamic selection, thereby reducing static background and repetitive visual information. 3) Extract historical information related to task instructions from intermediate candidate visual tokens through instruction-guided aggregation, and aggregate multiple candidate visual tokens into compact long-term memory perceptual anchors; 4) Cross-attention retrieval of hierarchical memory sequences is performed through action hidden states, and action experts are adaptively modulated using scale and offset parameters to avoid directly extending the context length of the visual language model; 5) It can improve the robot's ability to recognize historical operation results, changes in object state and task execution progress, reduce the impact of visual occlusion and state confusion on action generation, and improve the execution accuracy and stability of tasks that rely on robot control for a long time. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments, so that those skilled in the art can understand and implement the present invention. It should be noted that the following embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or adjustments made to the network structure, token quantity, memory capacity, sampling interval, and action generation method without departing from the concept of the present invention should all be included within the scope of protection of the present invention.

[0038] Example 1 Implementation, for example Figure 1 As shown, this embodiment provides a robot manipulation method based on hierarchical compact visual motion memory. This method is applicable to robot systems with visual sensors, robot state feedback interfaces, and action execution interfaces. Inputs include the current visual observation. Task instructions and robot status information The output is a sequence of actions used to control the robot to perform manipulation tasks. The manipulation tasks can be grasping, placing, counting, or combined operations with occlusion, repeated observations, and long-term temporal dependencies.

[0039] At time step The robot acquires current visual observations Task instructions Robot status information Visual tokens Visual historical information at the current time step is written into the historical observation sequence; the current visual observation is... With task instructions Multimodal context encoding is performed to generate the hidden motion state for initializing the motion expert; among which, task instructions... Simultaneously, robot status information serves as a text query condition for instruction-guided aggregation. Visual observation serves as the motion generation condition for iterative decoding by motion experts. Robot state information can be provided by a monocular camera, a multi-view camera, a wrist camera, or a combination thereof; This can include joint position, joint velocity, end effector pose, gripper state, and other information that describes the robot's current execution state. First, visual observation... and task instructions The process involves encoding the information and then writing the visual-motor history into short-term, medium-term, and long-term memory based on the time distance between the historical information and the current moment. The three levels of memory are then retrieved and modulated during the action generation stage.

[0040] like Figure 1 As shown, this invention provides a robot manipulation method based on hierarchical compact visual-motor memory, the implementation of which is as follows: S1. Obtain visual motion history information: at time step Obtain the robot's current visual observation Task instructions and robot status information and utilize visual encoders Current visual observation Image mapped to a set of visual tokens , where each visual token All include time location codes and spatial location coding ; In this embodiment, the visual encoder It can include an image preprocessing layer, a visual backbone network, and a visual projection layer. Visual encoder By performing block segmentation or other spatial feature extraction operations on the input image, a set of visual tokens arranged according to a two-dimensional spatial grid is obtained:

[0041] in, Indicates the first Frame space location Visual tokens at the location, and These represent the height and width of the visual feature grid, respectively.

[0042] To retain visual tokens The temporal order and spatial location within the historical observation sequence for each visual token Configure time location encoding and spatial location coding : .

[0043] In this embodiment, a visual language model with 3 billion parameters is used as the visual language backbone, and a flow-matching action expert with 300 million parameters is used to generate continuous action sequences. Visual encoder The image is kept frozen during training. The input visual observations have a resolution of 224×224, and each frame of visual observation is encoded as 256 visual tokens. Visual tokens Arranged in a 16×16 spatial grid, each visual token The feature dimension is 2048. Task instructions. The maximum text length is set to 64 tokens, and the motion expert's motion prediction length is set to 20 steps. Starting from the initial frame, every interval... An environmental step for current visual observation Perform a historical sampling and write it into memory.

[0044] S2. Constructing hierarchical visual motion memory: During continuous robot operation, the historical observation sequence is... Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints It is also divided into short-term memory based on time scale. Mid-term memory and long-term memory The implementation method is as follows: From the initial time step to the current time step The historical observation sequence is denoted as Among them, current visual observation After visual encoder Visual tokens obtained from mapping Visual historical information, as the current time step, is written into the historical observation sequence; During the continuous operation of the robot, the historical observation sequence will be used. Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints :

[0045] in, Indicates by task instructions The memory compression map consists of write, select, aggregate, and update operations. Indicates the total visual token capacity. This represents the feature dimension of the unified memory token; Representing compact visual memory Based on time scale, it is divided into short-term memory. Mid-term memory and long-term memory ,satisfy:

[0046]

[0047] in, Indicates time step A hierarchical and compact set of visual memory, Represents hierarchical index The corresponding subset of memories, Represents hierarchical index The corresponding capacity limit, Indicates short-term memory levels. Indicates the intermediate memory level. Indicates the long-term memory hierarchy. , and They represent short-term memory. Maximum capacity, intermediate memory Maximum capacity and long-term memory Maximum capacity, This indicates the total visual token capacity.

[0048] In this embodiment, the historical observation sequence It is converted into a compact visual memory representation under a fixed capacity constraint, and the memory hierarchy is divided according to the time distance between historical information and the current moment. Specifically, it is divided into memory levels from the initial time step to the current time step. The historical observation sequence is denoted as Through memory compression mapping Obtain a compact visual memory representation Representing compact visual memory Classified as short-term memory Mid-term memory and long-term memory .

[0049] Short-term memory It primarily stores visual information recent to the present moment to preserve details of recent operations; mid-term memory... It primarily preserves key motion cues selected through spatiotemporal dynamics to maintain information on changes between continuous observations; long-term memory. It primarily stores task-related perceptual anchors aggregated under instruction guidance to represent historical semantics over a longer timescale. The total capacity of the three-level memory consists of the total visual token capacity. This constraint ensures that the size of the historical memory accessed by the motion expert remains bounded regardless of how long the robot performs the task.

[0050] S3, Progressive Visual-Motor Memory Compression: This method compresses data written into short-term memory. Visual tokens Perform parameterless average pooling to generate short-term visual tokens; when short-term memory... Not exceeding its capacity At that time, short-term visual tokens are written into short-term memory in chronological order. When short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory And filter out spatiotemporal redundant tokens; during the middle period memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. ; For writing to short-term memory Visual tokens Perform parameterless average pooling to generate short-lived visual tokens, including the following steps: For writing to short-term memory The Frame visual token Divided into Non-overlapping local spatial regions And perform average pooling within each local spatial region to generate short-lived visual tokens:

[0051] in, This represents the first value obtained after average pooling. A short-term visual token Indicates the first A local spatial region, This indicates the number of visual tokens contained in the area. Indicates the first Frame space location Visual tokens at the location; The expression for the spatiotemporal location encoding corresponding to the short-term visual token is as follows:

[0052] in, Indicates the first Frame number Each pooled region corresponds to the spatiotemporal location encoding of the visual token. Indicates the first The location of each pooled region within the compressed spatial grid.

[0053] In this embodiment, the spatial visual token of the current frame is divided into non-overlapping local spatial regions, and average pooling is performed in each local spatial region to generate a short-term visual token to be written into short-term memory. Specifically, the first... Frame visual token Divided into Non-overlapping local spatial regions For each local spatial region, the visual tokens are averaged and pooled in a dimension-wise manner to obtain short-term visual tokens; the pooled short-term visual tokens are then associated with their corresponding temporal positions and compressed spatial grid positions.

[0054] Average pooling is a parameterless operation, introducing no additional learnable parameters and requiring no manual annotation of local spatial regions. When short-term memory... Capacity not reached At that time, the generated short-term visual tokens are written into short-term memory in chronological order. When short-term memory reaches its capacity limit, spatiotemporal dynamic selection is triggered.

[0055] In this embodiment, when short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory This includes the following steps: When short-term memory Exceeding its capacity At that time, for the current overflow frame and its adjacent reference frame Calculate visual tokens for the same spatial region Time series change score:

[0056] in, Indicates the first The temporal variation score of the visual token corresponding to each local spatial region. Indicates the current overflow frame The Middle Short-term visual tokens corresponding to local spatial regions Indicates adjacent reference frames The Middle Short-term visual tokens corresponding to local spatial regions This represents a preset constant to prevent the denominator from being zero; The local spatial region with the highest temporal change score is identified as the current overflow frame. Time change anchor point :

[0057] in, This operation represents the operation of returning the index of the region corresponding to the maximum temporal change score from all local spatial regions; Time-varying anchor points Calculate the local spatial region Chebyshev distance between the corresponding visual token and the time-varying anchor point:

[0058] in, Represents a local spatial region Anchor points of time change Chebyshev distance between them Represents a local spatial region Row coordinates in a compressed space grid Anchor points representing time changes Row coordinates in a compressed space grid Represents a local spatial region Column coordinates in a compressed space grid. Anchor points representing time changes Column coordinates in a compressed spatial grid; Domain-based reweighting of temporal variation scores based on Chebyshev distance:

[0059] in, Represents the i-th element after spatial neighborhood reweighting. The spatiotemporal saliency score of a local spatial region Anchor points representing time changes The radius of the surrounding spatial neighborhood, This represents the enhancement coefficient of the neighborhood region. This represents the scaling factor for the area outside the neighborhood; Based on the temporal change scores after domain reweighting, the top M key visual tokens are selected. Write into intermediate memory :

[0060]

[0061] in, Indicates the current overflow frame The mid-term visual token set obtained after spatiotemporal dynamic selection Key visual token index, Indicates an overflow frame Write into intermediate memory Key visual tokens Indexed collection, Represents the real number field. Represents the dimension of visual features.

[0062] When short-term memory Exceeding capacity When comparing the current overflow frame adjacent reference frames Visual tokens in the same spatial location The semantic change score is calculated, and a medium-term memory token is selected based on spatial neighborhood relationships to perform spatiotemporal dynamic selection and write it into medium-term memory. Specifically, let the current short-term memory be... The frame where the overflow occurred is The reference frame adjacent to it is When performing spatiotemporal dynamic selection, the current overflow frame and adjacent reference frames All have been written into short-term memory Therefore, the selection process only uses the visual tokens that are already cached at the current time. Future frame information is not accessed. For local spatial regions... Calculate visual tokens for the same spatial location in two frames. The cosine variation distance between them yields visual tokens for the same spatial region. The time series variation score between them, where the time series variation score The larger the score, the more significant the semantic change in the spatial region between adjacent observations; the temporal change score... The largest spatial region is determined as the current overflow frame. The time-varying anchor point; to preserve local spatial continuity while retaining regions with significant time variations, the computational region... The Chebyshev distance between the time-varying anchor points and the time-varying points is used to reweight the time-varying scores based on the Chebyshev distance. This neighborhood reweighting prioritizes regions adjacent to significantly changed regions and exhibiting high semantic changes, while suppressing isolated noise far from the changed regions. The highest-scoring time-varying score is selected based on the neighborhood reweighted time-varying scores. Visual token Selected visual token Write it into medium memory according to its frame index and spatial location encoding. ; during the middle period When the capacity limit is reached, an instruction is triggered to guide aggregation.

[0063] In this embodiment, during the intermediate memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. This includes the following steps: When intermediate memory Exceeding its capacity At that time, for the frame to be migrated Based on the highest M key visual tokens Calculate each candidate visual token retained by spatiotemporal dynamic selection. With task instructions The correlation score between the text and images:

[0064] in, Represents intermediate memory Candidate visual tokens With task instructions The image-text relevance score between query vectors. and These represent the visual projection matrix and the speech projection matrix, respectively. Represents the set of candidate visual tokens. Represents a text query vector. Indicates the candidate visual token index. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; The image-text relevance score is fused with the spatiotemporal saliency score obtained during the spatiotemporal dynamic selection phase to obtain a comprehensive selection score, and the L candidate visual tokens with the highest comprehensive scores are selected:

[0065] in, This represents the set of visual token indices that participate in long-term aggregation. and These represent the image-text relevance scores, respectively. Spatiotemporal saliency score The control coefficient, This indicates that the candidate visual tokens with the highest overall score are selected. Operations corresponding to the index of each token. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; Normalization with a temperature parameter is performed on the L candidate visual tokens to obtain the aggregate weights:

[0066] in, Indicates the first The aggregate weight of each candidate visual token. Indicates temperature parameter, Represents the visual token index set Elements in; The candidate visual tokens are weighted and summed according to the aggregation weights to generate perceptual anchors for long-term memory:

[0067] in, This represents the perceptual anchor point that is written into long-term memory. Represents the real number field. Indicates the dimension of visual features; Write the perceptual anchors of long-term memory into long-term memory .

[0068] When intermediate memory Exceeding capacity At that time, the image-text relevance score is calculated on the candidate visual tokens that have been retained in a spatiotemporal dynamic selection, and the task-related candidates are aggregated into long-term memory perceptual anchors. The aggregation is guided by the execution instruction and written into long-term memory. Specifically, for the frame to be transferred and its role in intermediate memory Candidate visual token set First, from the task instructions After filtering out padding tokens and invalid text tokens, the remaining valid text tokens are aggregated to obtain the text query vector. ; through visual projection matrix and language projection matrix Combine candidate visual tokens with text query vectors Mapped to the same feature space, and L2 normalized respectively, by calculating the first... Candidate visual tokens With task instructions The cosine similarity between the two is used to obtain the image-text relevance score, which is only retained in a spatiotemporally dynamic selection. The calculation is performed on each candidate visual token, without recalculating all the original visual tokens that have already been filtered by the spatiotemporal dynamic selection, thus reducing the computational load in the long-term aggregation stage. The image-text relevance score is fused with the reweighted significance score obtained in the spatiotemporal dynamic selection stage to obtain a comprehensive selection score, and the one with the highest comprehensive score is selected. 1. Select candidate visual tokens; 2. Perform Softmax normalization with temperature parameter on the selected candidate visual tokens to obtain their aggregate weights; 3. Perform weighted summation on the candidate visual tokens according to the aggregate weights to generate long-term memory perceptual anchors.

[0069] In one implementation, the long-term memory perceptual anchor not only stores the aggregated visual features, but also associates its source frame index, the spatial location index participating in the aggregation, and the valid token mask; in another implementation, the spatiotemporal location encoding has already injected visual features during the visual token formation stage, and the long-term memory perceptual anchor is directly used as a long-term memory token containing spatiotemporal semantics.

[0070] S4, Visual-Motor Memory Injection: The short-term, medium-term, and long-term memories processed in S3 are concatenated and converted into a memory token sequence, which is then injected into the motion expert. Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Adaptive modulation is implemented as follows: The short-term, medium-term, and long-term memories processed by S3 are concatenated and mapped into a hierarchical memory sequence. Action expert Hidden state of layer actions As a query, hierarchical memory sequence Perform cross-attention as keys and values ​​to obtain a memorized conditional representation:

[0071] in, For the purpose of memory condition representation, , and These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. Represents the attention feature dimension. This represents the memory mask generated based on the valid memory token; Representing memory conditions Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Perform adaptive modulation:

[0072]

[0073] in, This indicates the hidden state of actions modulated by memory. This indicates the hidden state of the next level of the motion expert. Indicates the gating coefficient. This indicates the current denoising time step for the motion expert.

[0074] In this embodiment, the three-level memory is mapped to a hierarchical memory sequence of uniform width. Cross-attention is performed using the motion expert's hidden motion state as the query, and the retrieval results are converted into scale and offset parameters to perform three-level memory retrieval and adaptive motion modulation. Specifically, short-term memory, medium-term memory, and long-term memory are mapped to the motion expert's memory width, respectively. They are then pieced together in a short-term, medium-term, and long-term order to form a hierarchical memory sequence. :

[0075] in, , and These represent projection operations that map different memory levels to a uniform feature width; Action Expert Hidden state of layer actions As a query, hierarchical memory sequence Perform cross-attention on the keys and values ​​to obtain a memory conditional representation: Among them, memory mask Used to mask invalid, filled, or unwritten memory locations, allowing motion experts to access only valid historical visual information.

[0076] In this embodiment, the memory-based cross-attention mechanism includes four query heads and one key-value head, with each attention head having a dimension of 256. The memory conditions are represented... Mapped to scale parameters through modulation projection layer and offset parameters Using scale parameters and offset parameters Hidden state of action experts Perform adaptive modulation:

[0077]

[0078] in, This represents the mapping of memory condition representations to modulation projection matrices with scale and offset parameters. This represents the bias vector of the modulated projection. This indicates the root mean square normalization operation. This represents element-wise multiplication, with a scale parameter. The offset parameter is used to adjust the response scale of the hidden state of an action based on historical memory. Feature offsets corresponding to historical memories are used to incorporate these offsets. The modulated action hidden state is input into the action expert's action generation layer and updated using residuals. By controlling the gating coefficient to manage the influence of historical memory on the current action state, the model can gradually introduce historical information according to the current action generation stage, reducing the excessive interference of historical features on the current observation.

[0079] S5. Output result: Hides state using modulated motion. Combined with current visual observations Task instructions and robot status information The initial noisy action sequence is iteratively decoded to output the action sequence used to control the robot. After the action is executed, the new observation is added to the historical observation sequence to achieve continuous memory updates.

[0080] In this embodiment, the modulated motion hiding state is utilized. Combined with current visual observations Task instructions and robot status information Iteratively decode the initial noisy action sequence to output the robot action sequence. The new observations after execution are fed back to the historical observation sequence to iteratively decode the action sequence and update the memory. Specifically, the action hidden state of the adaptive memory modulation mechanism is then used. Input stream matching action expert, starting with an initial noisy action sequence, combined with current visual observations Task instructions and robot status information Iterative updates are performed at multiple denoising time steps.

[0081] No. The action update for each denoising time step can be represented as:

[0082] in, Indicates the first A denoised time step action sequence. This represents the motion hidden state used for motion velocity field prediction after adaptive memory modulation. Indicates the first A denoised time step action sequence. Indicates the current denoising time. This represents the motion velocity field predicted by motion experts. This indicates the step size between adjacent denoising time steps. This indicates the robot's status information.

[0083] After a preset number of iterations, the robot's action sequence is output. :

[0084] in, and These represent the second-to-last and last actions in the output action sequence, respectively. During online closed-loop execution, visual observations are provided at 20 frames per second. Each action inference executes the first 16 steps of the action sequence, then reacquires visual observations and robot state. Newly acquired observations are added to the historical observation sequence, and the Level 3 visual motion memory is updated according to S1 to S5.

[0085] In summary, this invention acquires current visual observations, task instructions, and robot state, generates visual tokens with spatiotemporal position encoding using a visual encoder, and constructs a three-level memory system—short-term memory, medium-term memory, and long-term memory—with a fixed token capacity. Short-term memory aggregates local visual features through a parameterless pooling mechanism, forming a global perception foundation for current action decisions while preserving the overall details of the original features. Medium-term memory uses a spatiotemporal dynamic selection module to retain key cue tokens based on inter-frame semantic changes and intra-frame spatial neighborhood consistency, while filtering out spatiotemporally redundant tokens. Long-term memory utilizes an instruction-guided aggregation module to select visual tokens related to task instructions from medium-term candidate tokens and aggregate them into perception anchors for long-term storage. During action decoding, an adaptive modulation mechanism retrieves historical information from the three-level memory and converts the retrieval results into scale and offset parameters to modulate the hidden state of the action expert, thereby guiding the generation of action sequences. This invention can reduce visual historical information redundancy and computational overhead while avoiding the loss of key information, improving the robot's performance in tasks with long-term dependencies and state confusion.

[0086] Example 2 In this embodiment, the visual encoder The text encoder, memory projection layer, and action expert can employ pre-trained parameters, joint-trained parameters, or partially frozen parameters depending on the specific task. Cosine variation distance, spatial neighborhood reweighting, Top-M selection, image-text relevance calculation, Top-L selection based on comprehensive score, and Softmax weighted aggregation in spatiotemporal dynamic selection can serve as memory compression mappings. The operations described above can be performed directly during inference or used in parameter optimization during model training as differentiable approximations or through-the-loop estimations.

[0087] In one implementation, the action generation model and memory modulation module are trained using the following comprehensive objective:

[0088]

[0089]

[0090] in, Represents the overall objective function. This indicates the loss in robot action prediction. This represents the auxiliary loss used to constrain memory retrieval, image-text relevance, or memory aggregation stability. This indicates the weight of the corresponding loss term. This represents the robot's actual movement trajectory. Indicates random sampling noise. This represents the time variable during the action generation process. This represents the intermediate motion state obtained by fusing real motion with noise. This represents the model's input condition information, including visual observations, language commands, and historical memory features. This indicates the direction of the action change output by the action prediction network. The target motion direction is represented by the L2 error between the predicted motion direction and the target motion direction. By minimizing the L2 error between the predicted motion direction and the target motion direction, the model learns an accurate robot motion generation strategy. Indicates the first The memory features at each time step are represented. This indicates the characteristics of historical memory at the previous time step. This represents the length of the memory sequence. The auxiliary loss reduces drastic changes in memory representations at consecutive time steps by constraining the distance between memory features at adjacent time steps, enabling the memory module to maintain temporal continuity and stability, and improving the ability of historical information to assist in robot action prediction.

[0091] in, Represents the overall objective function. This indicates the loss in robot action prediction. This represents the auxiliary loss used to constrain memory retrieval, image-text relevance, or memory aggregation stability. This indicates the weight of the corresponding loss term.

[0092] In another implementation, the pooling region of short-term memory can be a regular grid, an overlapping grid, an adaptive region, or a region partitioned by the visual encoder; the similarity in spatiotemporal dynamic selection can be cosine distance, normalized inner product, or other distances that can measure the semantic changes of adjacent visual tokens; the text query in instruction-guided aggregation can be average pooling of effective text tokens, attention pooling, special text tokens in the last layer, or other language feature aggregation methods. Any implementation that can achieve hierarchical compression and action modulation of historical visual information within a fixed memory capacity falls within the scope of this invention.

[0093] In another implementation, the motion expert can employ a diffusion-based motion generation model, a flow-matching motion generation model, an autoregressive motion generation model, or other continuous motion sequence generation models. Adaptive memory modulation can be inserted into one or more network layers of the motion expert, and scale and offset parameters can be applied to the normalized hidden state, residual branch, feedforward branch, or attention branch.

[0094] In summary, this invention achieves progressive compression and task-related retrieval of historical visual motion information with a fixed visual token capacity by using parameterless pooling of short-term memory, spatiotemporal dynamic selection of medium-term memory, instruction-guided aggregation of long-term memory, and adaptive memory modulation by motion experts. It is suitable for robot manipulation tasks with long-term dependencies, visual occlusion, state confusion, and repetitive interactions.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Equivalent changes or substitutions made by those skilled in the art to the number of memory levels, token capacity allocation, pooling region division, score fusion method, aggregation weight calculation method, modulation insertion position, and action generation network without departing from the spirit and substance of the present invention should all be covered within the scope of protection of the present invention.

Claims

1. A robot manipulation method based on hierarchical compact visual-motor memory, characterized in that, Includes the following steps: S1. Obtain visual motion history information: at time step Obtain the robot's current visual observation Task instructions and robot status information and utilize visual encoders Current visual observation Image mapped to a set of visual tokens , where each visual token All include time location codes and spatial location coding ; S2. Constructing hierarchical visual motion memory: During continuous robot operation, the historical observation sequence is... Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints It is also divided into short-term memory based on time scale. Mid-term memory and long-term memory ; S3, Progressive Visual-Motor Memory Compression: This method compresses data written into short-term memory. Visual tokens Perform parameterless average pooling to generate short-lived visual tokens; When short-term memory Not exceeding its capacity At that time, short-term visual tokens are written into short-term memory in chronological order. When short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory And filter out spatiotemporal redundant tokens; during the middle period memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. ; S4, Visual-Motor Memory Injection: The short-term, medium-term, and long-term memories processed in S3 are concatenated and converted into a memory token sequence, which is then injected into the motion expert. Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Perform adaptive modulation; S5. Output result: Hides state using modulated motion. Combined with current visual observation Task instructions and robot status information The initial noisy action sequence is iteratively decoded to output the action sequence used to control the robot. After the action is executed, the new observation is added to the historical observation sequence to achieve continuous memory updates.

2. The robot control method based on hierarchical compact visual-motor memory according to claim 1, characterized in that, The construction of hierarchical visual-motor memory includes the following steps: From the initial time step to the current time step The historical observation sequence is denoted as Among them, current visual observation After visual encoder Visual tokens obtained from mapping Visual historical information, as the current time step, is written into the historical observation sequence; During the continuous operation of the robot, the historical observation sequence will be used. Through memory compression mapping Transformed into a compact visual memory representation under fixed capacity constraints : in, Indicates by task instructions The memory compression map consists of write, select, aggregate, and update operations. Indicates the total visual token capacity. This represents the feature dimension of the unified memory token; Representing compact visual memory Based on time scale, it is divided into short-term memory. Mid-term memory and long-term memory ,satisfy: in, Indicates time step A hierarchical and compact set of visual memory, Represents hierarchical index The corresponding subset of memories, Represents hierarchical index The corresponding capacity limit, Indicates short-term memory levels. Indicates the intermediate memory level. Indicates the long-term memory hierarchy. , and They represent short-term memory. Maximum capacity, intermediate memory Maximum capacity and long-term memory Maximum capacity, This indicates the total visual token capacity.

3. The robot control method based on hierarchical compact visual-motor memory according to claim 1, characterized in that, The writing to short-term memory Visual tokens Perform parameterless average pooling to generate short-lived visual tokens, including the following steps: For writing to short-term memory The Frame visual token Divided into Non-overlapping local spatial regions And perform average pooling within each local spatial region to generate short-lived visual tokens: in, This represents the first value obtained after average pooling. A short-term visual token Indicates the first A local spatial region, This indicates the number of visual tokens contained in the area. Indicates the first Frame space location Visual tokens at the location; The expression for the spatiotemporal location encoding corresponding to the short-term visual token is as follows: in, Indicates the first Frame number Each pooled region corresponds to the spatiotemporal location encoding of the visual token. Indicates the first The location of each pooled region within the compressed spatial grid.

4. The robot control method based on hierarchical compact visual-motor memory according to claim 1, characterized in that, The short-term memory Exceeding its capacity At that time, the spatiotemporal dynamic selection module is used to filter out key visual tokens. Write into intermediate memory This includes the following steps: When short-term memory Exceeding its capacity At that time, for the current overflow frame and its adjacent reference frame Calculate visual tokens for the same spatial region Time series change score: in, Indicates the first The temporal variation score of the visual token corresponding to each local spatial region. Indicates the current overflow frame The Middle Short-term visual tokens corresponding to local spatial regions Indicates adjacent reference frames The Middle Short-term visual tokens corresponding to local spatial regions This represents a preset constant to prevent the denominator from being zero; The local spatial region with the highest temporal change score is identified as the current overflow frame. Time change anchor point : in, This operation represents the operation of returning the index of the region corresponding to the maximum temporal change score from all local spatial regions; Time-varying anchor points Calculate the local spatial region Chebyshev distance between the corresponding visual token and the time-varying anchor point: in, Represents a local spatial region Anchor points of time change Chebyshev distance between them Represents a local spatial region Row coordinates in a compressed space grid Anchor points representing time changes Row coordinates in a compressed space grid Represents a local spatial region Column coordinates in a compressed space grid. Anchor points representing time changes Column coordinates in a compressed spatial grid; Domain-based reweighting of temporal variation scores based on Chebyshev distance: in, Represents the i-th element after spatial neighborhood reweighting. The spatiotemporal saliency score of a local spatial region Anchor points representing time changes The radius of the surrounding spatial neighborhood, This represents the enhancement coefficient of the neighborhood region. This represents the scaling factor for the area outside the neighborhood; Based on the temporal change scores after domain reweighting, the top M key visual tokens are selected. Write into intermediate memory : in, Indicates the current overflow frame The mid-term visual token set obtained after spatiotemporal dynamic selection Key visual token index, Indicates an overflow frame Write into intermediate memory Key visual tokens Indexed collection, Represents the real number field. Represents the dimension of visual features.

5. The robot control method based on hierarchical compact visual-motor memory according to claim 1, characterized in that, The intermediate memory Exceeding its capacity At that time, the aggregation module is guided by instructions to select candidate visual tokens. In and Task Instructions Related tokens are aggregated into perceptual anchors and written into long-term memory. This includes the following steps: When intermediate memory Exceeding its capacity At that time, for the frame to be migrated Based on the highest M key visual tokens Calculate each candidate visual token retained by spatiotemporal dynamic selection. With task instructions The correlation score between the text and images: in, Represents intermediate memory Candidate visual tokens With task instructions Image-text relevance score between query vectors. and These represent the visual projection matrix and the speech projection matrix, respectively. Represents the set of candidate visual tokens. Represents a text query vector. Indicates the candidate visual token index. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; The image-text relevance score is fused with the spatiotemporal saliency score obtained during the spatiotemporal dynamic selection phase to obtain a comprehensive selection score, and the L candidate visual tokens with the highest comprehensive scores are selected: in, This represents the set of visual token indices that participate in long-term aggregation. and These represent the image-text relevance scores, respectively. Spatiotemporal saliency score The control coefficient, This indicates that the candidate visual tokens with the highest overall score are selected. Operations corresponding to the index of each token. Indicates an overflow frame Write into intermediate memory Key visual tokens Index set; Normalization with a temperature parameter is performed on the L candidate visual tokens to obtain the aggregate weights: in, Indicates the first The aggregate weight of each candidate visual token. Indicates temperature parameter, Represents the visual token index set Elements in; The candidate visual tokens are weighted and summed according to their aggregation weights to generate perceptual anchors for long-term memory. in, This represents the perceptual anchor point that is written into long-term memory. Represents the real number field. Indicates the dimension of visual features; Write the perceptual anchors of long-term memory into long-term memory .

6. The robot control method based on hierarchical compact visual-motor memory according to claim 1, characterized in that, S4 includes the following steps: The short-term, medium-term, and long-term memories processed by S3 are concatenated and mapped into a hierarchical memory sequence. Action expert Hidden state of layer actions As a query, hierarchical memory sequence Perform cross-attention as keys and values ​​to obtain a memorized conditional representation: in, For the purpose of memory condition representation, , and These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. Represents the attention feature dimension. This represents the memory mask generated based on the valid memory token; Representing memory conditions Scale parameters are generated through linear projection. and offset parameters and using scale parameters and offset parameters Hide state of actions Perform adaptive modulation: in, This indicates the hidden state of actions modulated by memory. This indicates the hidden state of the next level of the motion expert. Indicates the gating coefficient. This indicates the current denoising time step for the motion expert.

Images