Dynamic memory augmentation method and system for visual-linguistic-action model and readable storage medium

Abstract

The application provides a dynamic memory enhancement method and system for a visual-language-action model and a storage medium, and solves the problems of short-sighted memory, memory pollution and inability to dynamically adjust the fusion weight of the existing VLA model in a long-distance task. The application comprises: constructing a working memory and retrieving historical memory from a perception-cognition-reward memory bank; using a gating network with trainable parameters to adaptively weight and fuse the current memory and the historical memory to generate an enhanced working memory; using a progress evaluator to calculate a task progress difference value based on the enhanced working memory to generate a single-step dense reward; using the reward as a merit function to perform online policy optimization on the gating network parameters; and when the memory bank is full, merging adjacent entries based on the weighted indicators of feature similarity and reward similarity; and finally outputting a motion sequence by a motion expert network. The application can be used for robot long-time sequence task learning and adaptive control in a dynamic environment.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence, visual-language-action models, and dynamic memory enhancement technologies, and particularly to a dynamic memory enhancement method for visual-language-action models. Background Technology

[0002] Vision-Language-Action (VLA) models have made significant progress in the field of embodied intelligence and have become a general paradigm for robot learning. However, existing VLA models face serious memory mechanism deficiencies in long-range tasks and dynamic environments, and their practical deployment suffers from the following main technical shortcomings: First, there are the challenges of memory shortsightedness and non-Markovian challenges. Mainstream VLA models, such as OpenVLA and RT-2, typically rely on the current single-frame image for action prediction, lacking an effective long-term memory mechanism and struggling to handle long-term, time-dependent tasks. For example, in button operation tasks, there is almost no visual difference before and after pressing the button, making it difficult for models lacking temporal memory to determine whether the action has been completed. To address this issue, Hao Shi et al. designed a "perceptual-cognitive memory bank" by drawing inspiration from the human brain's cognitive mechanisms, proposing a memory-enhanced VLA model—MemoryVLA. This model effectively alleviates the memory shortsightedness problem of VLA models when facing long-range visual tasks by introducing an external memory mechanism. However, in actual deployment, the parameters of MemoryVLA are completely frozen, and its memory fusion weights cannot be dynamically adjusted with the environment. When there is insufficient reliability between historical state information and current observation data, the fused information carries errors; and the information carrying errors participates in iterative fusion as new historical state information, causing the errors in the system state information bank to continuously accumulate and expand, forming memory pollution.

[0003] Second, there is the issue of memory invalidation due to distribution shift. Mainstream VLA models are trained through supervised fine-tuning or training-time reinforcement learning, requiring explicit fine-tuning phases, manual intervention, or controlled data acquisition. Existing methods are still not suitable for challenging simulation or physical world deployment scenarios. When the environment changes, the model's historical memory becomes semantically offset from the current observation, and the original memory fusion weights become invalid, leading to a significant decline in model performance. To address this, Changyu Liu et al. proposed a test-period reinforcement learning VLA model—TT-VLA—which adapts to environmental changes through online optimization during the testing phase. However, TT-VLA lacks long-term temporal historical memory, and its progress evaluation relies on original image observations, making it difficult to handle complex tasks such as those with no significant state changes.

[0004] Third, the memory merging and forgetting mechanisms have flaws. Existing memory consolidation modules typically rely solely on feature cosine similarity for merging decisions and employ a mean-based merging strategy. This fails to distinguish between high-value and low-value memories, leading to the loss of crucial operational details during the merging process, while redundant and erroneous exploration trajectories are retained, resulting in a decline in the quality of the memory bank.

[0005] In summary, existing VLA models lack an adaptive memory mechanism that can both maintain long-term memory and dynamically adjust fusion weights according to the environment; there is no effective time-aware progress evaluation signal to drive online optimization of memory weights; and the memory consolidation mechanism cannot selectively retain or forget based on task value. Therefore, there is an urgent need for a memory enhancement method that can dynamically adjust memory fusion weights, generate effective progress reward signals, and distinguish memory value during the testing phase, in order to improve the VLA model's performance on long-range tasks in unknown dynamic environments. Summary of the Invention

[0006] This invention addresses several issues in existing technologies, including: existing models relying on the current single-frame image for action prediction, lacking long-term memory mechanisms; memory fusion weights in memory-enhancing models being completely frozen during testing and unable to dynamically adjust with the environment; semantic shifts between the model's historical memory and current observations when the environment changes; and existing memory consolidation mechanisms relying solely on feature cosine similarity for merging, failing to distinguish between high-value and low-value memories. Therefore, this invention proposes a dynamic memory enhancement method for vision-language-action models, comprising: Step 1: Obtain the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as the query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on the sinusoidal time step position encoding. Step 2: Input the working memory and historical memory into a gating network with trainable parameters, calculate the fusion weights through the gating network, and perform weighted fusion of the current features and historical features to output the enhanced working memory; Step 3: Input the enhanced working memory into the frozen progress evaluator to calculate the current task progress, generate a single-step dense reward based on the progress difference between adjacent time steps, and cache the memory tuple containing the single-step dense reward into the perception-cognition-reward memory bank. Step 4: Using single-step dense reward as the advantage function, the trainable parameters in the gated network are updated with gradients using a policy optimization algorithm. When the capacity of the perception-cognition-reward memory bank reaches its limit, adjacent memory entries are merged with reward weighting based on a weighted index of feature similarity and reward similarity. Step 5: Input the enhanced working memory into the action expert network and output the predicted action sequence.

[0007] Furthermore, in step one, The process of acquiring the current observation data and natural language instructions, extracting perceptual tokens and cognitive tokens to construct working memory specifically involves: inputting the observation data into a pre-trained visual encoder to extract the original visual features, and then reducing the dimensionality of the data using a perceptual compression module to obtain the perceptual token; projecting the original visual features into the embedding space of a large language model, concatenating them with the word vectors of the natural language instructions, and then inputting them into the large language model to output the cognitive token; and concatenating the perceptual token and the cognitive token to form the working memory at the current moment. The process of using working memory as a query vector and retrieving historical memories from the perception-cognition-reward memory bank based on time position encoding specifically involves: adding the historical perception token and historical cognition token stored in the perception-cognition-reward memory bank to the sinusoidal time step position encoding as the key; using the perception token and cognition token in the current working memory as the query; calculating attention weights through a cross-attention mechanism; and retrieving the historical perception token, historical cognition token, and corresponding historical reward in a weighted manner.

[0008] Furthermore, in step two, The calculation of fusion weights through the gating network specifically involves: concatenating the current perception token with the historical perception token to form a joint perception feature, and concatenating the current cognition token with the historical cognition token to form a joint cognition feature; inputting the joint perception feature and the joint cognition feature into the gating network respectively, and after joint calculation by the low-rank adaptive matrix and the frozen weight matrix, outputting the perception gating vector and the cognition gating vector through the Sigmoid activation function; The weighted fusion of current and historical features to output enhanced working memory specifically involves using the perceptual gating vector and cognitive gating vector to multiply the current and historical features element-wise and sum them to obtain the enhanced perceptual token and the enhanced cognitive token.

[0009] Furthermore, The single-step intensive reward is the difference between the task progress at the current moment and the task progress at the previous moment.

[0010] Furthermore, in step four, The step of using single-step dense reward as the advantage function and employing a policy optimization algorithm to perform gradient updates on the trainable parameters in the gated network specifically includes: using a valueless proximal policy optimization algorithm, after a preset number of time steps K, performing a gradient update on the trainable parameters in the gated network using a loss function constructed based on single-step dense reward, while keeping the parameters of the visual encoder, large language model, and action expert network frozen.

[0011] Furthermore, in step four, The step of merging adjacent memory entries based on a weighted index of feature similarity and reward similarity when the capacity of the perception-cognition-reward memory bank reaches its limit is as follows: When the number of entries in the perception-cognition-reward memory bank exceeds a preset capacity limit, a merging index is calculated between adjacent memory entries. This merging index is a weighted sum of the feature cosine distance and the reward Manhattan distance. The calculation method for the merging index is as follows:

[0012] in, , Enhance working memory for adjacent memory items; , For the corresponding historical rewards; , Preset weighting coefficients; Select the pair of adjacent entries with the smallest merging index, calculate the weighted average weight based on their historical rewards, merge the features and rewards, and generate a new memory entry.

[0013] Furthermore, Based on its historical rewards, a weighted average weight is calculated, and the features and rewards are merged to generate a new memory entry. Specifically: The historical rewards of two adjacent entries to be merged are converted into merge weights using a Softmax function with a temperature coefficient. The calculation method is as follows:

[0014]

[0015] in, Temperature coefficient; The calculation is performed after merging and memorizing; the calculation method is as follows:

[0016] The calculation method for the merged reward is as follows:

[0017] Replace the two existing adjacent entries with the merged memory, merged reward, and merged timestamp as new entries.

[0018] Furthermore, The memory tuples stored in the perception-cognition-reward memory bank include enhanced working memory, corresponding single-step intensive rewards, and sinusoidal time-step position codes.

[0019] This invention proposes a dynamic memory enhancement system for a vision-language-action model, comprising: The working memory construction and retrieval module is used to acquire the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as a query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on time location encoding. The gated fusion module is used to input working memory and historical memory into a gated network with trainable parameters. The gated network calculates fusion weights and performs weighted fusion of current features and historical features to output enhanced working memory. The progress assessment and reward generation module is used to input the enhanced working memory into the frozen progress evaluator to calculate the current task progress, generate single-step dense rewards based on the progress difference between adjacent time steps, and cache the memory tuple containing the single-step dense rewards into the perception-cognition-reward memory bank. The online optimization and memory consolidation module is used to use single-step dense reward as the advantage function, and to perform gradient updates on the trainable parameters in the gated network using a policy optimization algorithm. When the capacity of the perception-cognition-reward memory bank reaches the upper limit, adjacent memory entries are merged based on a weighted index of feature similarity and reward similarity. The action generation module is used to input enhanced working memory into the action expert network and output predicted action sequences.

[0020] This invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the above-described method.

[0021] The beneficial effects of this invention are: This invention proposes a dynamic memory enhancement method, system, and storage medium for vision-language-action models, which has the following improvements: 1. To mitigate memory contamination and distribution shift issues, a gated network with trainable parameters is constructed. The single-step dense reward generated by the progress evaluator is used as a feedback signal. A policy optimization algorithm is employed to update the gated network parameters online, enabling the model to dynamically adjust the fusion weights of the current observation and historical memory during the testing phase. This allows the model to adapt to environmental changes and effectively suppress the performance degradation caused by memory contamination and distribution shift.

[0022] 2. To improve the success rate of long-range tasks, a single-step dense reward mechanism based on the progress difference between adjacent time steps is introduced. The task progress perception signal is introduced into the memory enhancement framework, providing a clear and physically meaningful optimization direction for the online optimization of gating fusion weights, thus solving the problem of the lack of effective progress evaluation signals in traditional methods.

[0023] 3. When the memory bank capacity reaches its limit, a weighted merging index of feature cosine distance and reward Manhattan distance is adopted, and reward-weighted merging is performed based on historical reward values. This ensures that high-reward, high-value memory entries dominate the merging process, while low-value or erroneous exploration memories are effectively forgotten, thereby alleviating memory contamination and feature dilution problems and ensuring that key operational details are preserved.

[0024] 4. Improved the computational efficiency and real-time deployment capability of the VLA model memory mechanism during the testing period, limiting the online fine-tuning range to the lightweight gated network LoRA parameters, eliminating the need to update the visual encoder, large language model and action generation head containing large-scale parameters, and meeting the requirements for low-latency real-time deployment. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Combination Figures 1-2 This invention proposes a dynamic memory enhancement method for a vision-language-action model, the method comprising: Step 1: Obtain the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as the query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on the sinusoidal time step position encoding. Furthermore, in step one, The process of acquiring the current observation data and natural language instructions, extracting perceptual tokens and cognitive tokens to construct working memory specifically involves: inputting the observation data into a pre-trained visual encoder to extract the original visual features, and then reducing the dimensionality of the data using a perceptual compression module to obtain the perceptual token; projecting the original visual features into the embedding space of a large language model, concatenating them with the word vectors of the natural language instructions, and then inputting them into the large language model to output the cognitive token; and concatenating the perceptual token and the cognitive token to form the working memory at the current moment. The process of using working memory as a query vector and retrieving historical memories from the perception-cognition-reward memory bank based on time position encoding specifically involves: adding the historical perception token and historical cognition token stored in the perception-cognition-reward memory bank to the sinusoidal time step position encoding as the key; using the perception token and cognition token in the current working memory as the query; calculating attention weights through a cross-attention mechanism; and retrieving the historical perception token, historical cognition token, and corresponding historical reward in a weighted manner.

[0029] Step one specifically involves: at any time t, the system acquires the current observation data, such as the RGB image I. t The system uses natural language instructions to construct working memory through a pre-trained visual language model and retrieves historical memories from the memory bank based on this. Specifically, it includes the following three sub-steps: S1.1: Constructing the Perceptual Token and Cognitive Token. The image is input into a pre-trained visual encoder to extract the raw visual features. Then, a perceptual compression module is used to reduce the dimensionality of the visual features to obtain the perceptual token (p). t The token contains low-level visual details; the original visual features are mapped to the embedding space of the large language model through a projection layer, while the natural language instruction l is processed by a word segmenter to obtain word vectors. The two parts are then concatenated. The concatenated sequence is input into the large language model to obtain the cognitive token (c). t The token contains advanced semantic information.

[0030] S1.2: Construct working memory. The p obtained in S1.1... t With c t The data is pieced together to form the working memory set M at the current moment. wk ={p t ,c t This working memory corresponds to human short-term memory, representing only the current state and reasoning about the current moment, and does not include temporal dependence.

[0031] S1.3: Memory Retrieval Based on Time-Location Encoding. A "Perceptual-Cognitive-Reward Memory Bank (PCRMB)" is designed to record historical memories. Memory retrieval is then performed using current working memory M.wk The PCRMB is used as a query vector. Historical perceptual / cognitive tokens stored in the PCRMB serve as values, which are then encoded with a sinusoidal time step position as the key. Cross-attention weights are calculated using the current perceptual / cognitive token, and the most suitable historical entry perceptual token (H) is retrieved. t p ) / Cognitive Token (H t c ), and receive the corresponding reward r. t .

[0032] Step 2: Input the working memory and historical memory into a gating network with trainable parameters, calculate the fusion weights through the gating network, and perform weighted fusion of the current features and historical features to output the enhanced working memory; Furthermore, in step two, The calculation of fusion weights through the gating network specifically involves: concatenating the current perception token with the historical perception token to form a joint perception feature, and concatenating the current cognition token with the historical cognition token to form a joint cognition feature; inputting the joint perception feature and the joint cognition feature into the gating network respectively, and after joint calculation by the low-rank adaptive matrix and the frozen weight matrix, outputting the perception gating vector and the cognition gating vector through the Sigmoid activation function; The weighted fusion of current and historical features to output enhanced working memory specifically involves using the perceptual gating vector and cognitive gating vector to multiply the current and historical features element-wise and sum them to obtain the enhanced perceptual token and the enhanced cognitive token.

[0033] Furthermore, The single-step intensive reward is the difference between the task progress at the current moment and the task progress at the previous moment.

[0034] Step two specifically involves: the system receiving the current working memory set {p} output from step one. t ,c t} and historical feature set {H t p H t c The current observation and historical memory are adaptively fused through a dynamically gated network with trainable parameters. This step specifically includes the following two sub-steps: S2.1: Gating weight calculation. Current perceived p t With historical perception By splicing the data, we obtain joint perception [p] t H tp To achieve real-time updates, the pre-trained gated MLP weight matrix W0 is kept parameter-frozen and a low-rank adaptive matrix ΔW is injected. The joint features are then input into the multilayer perceptron to compute the gate weights, i.e., the perceptual gate vector g. t p With cognitive gating vector g t c .

[0035] S2.2: Constructing Enhanced Working Memory. Using the generated gating vector, weighted fusion of current and historical features outputs an enhanced perceptual token. t and Enhanced Cognitive Token t A set of.

[0036] Step 3: Input the enhanced working memory into the frozen progress evaluator to calculate the current task progress, generate a single-step dense reward based on the progress difference between adjacent time steps, cache the memory tuple containing the single-step dense reward into the perception-cognition-reward memory bank, and fine-tune the trainable parameters in the gating network online according to the preset update frequency using the single-step dense reward. Step three specifically involves using a pre-trained progress estimator. This involves calculating the progress of enhanced working memory, using the progress difference at different time steps as the intensive reward. Specifically, it includes the following four sub-steps: S3.1: Task progress calculation. A progress evaluation network is pre-trained using successful human demonstration trajectory data. Then freeze the parameters. At the current time step t, enhance the perception token. t Enhanced cognitive token t The natural language instructions are concatenated and input into the progress evaluation network. Progress is being made t , representing the percentage of the current state from the final target state described by the natural language instruction l. Using enhanced features instead of the original image for progress calculation can fully utilize the extracted semantic information, reduce computational latency, and meet real-time requirements.

[0037] S3.2: Dense Reward Calculation. Progress is calculated once at each time step, and the dense reward is constructed using this progress. The reward r at the current time t is... t This is a single-step intensive reward.

[0038] This dense reward has a clear physical meaning: (1) When r tWhen the value is greater than 0, it indicates that the action performed at time step t has made progress in the task, and the gating fusion weights are effective. (2) When r t When the value is approximately 0, it indicates that the system is in a stagnant state or has performed an invalid action, and the gating fusion weights should be adjusted. (3) When r t A value less than 0 indicates regression and a deviation in the gating weights.

[0039] S3.3: Cache memory entries. The calculated reward r... t With enhanced feature set t ={ t , t The time step position code TE(t) and the time step position code form a valuable memory tuple that is suspended in PCRMB.

[0040] S3.4: Fine-tune the gated LoRA parameters through backpropagation. This involves adjusting the dense reward r... t As a feedback signal, in the reward r t When the reward is positive, the model will slightly increase the probability of the corresponding action and the fusion weight; when the reward is negative, the model will decrease the probability of that strategy. An update frequency is set, triggering an update every K time steps, and gradient backpropagation only applies to trainable parameters.

[0041] Specifically, the progress estimation function is calculated as follows: Let the enhanced sensing token at time t be... Enhance cognitive tokens The text embedding of natural language instructions is E l Define the schedule evaluation network as follows: ( The formula for calculating progress is:

[0042] Output progress scalar ∈[0,1] represents the percentage of the current state that is far from the final target state described by the natural language instruction l.

[0043] The dense reward calculation method specifically involves using the progress scalars of adjacent time steps to calculate the dense reward at different time steps:

[0044] The memory tuple is specifically: at time t, t ={ t , tTo enhance memory, TE(t) is the sinusoidal time step position encoding, and the entry definition for PCRMB is as follows:

[0045] in, It is used to retrieve historical memories and rewards, thereby enabling memory fusion and consolidation.

[0046] Step 4: Input the enhanced working memory into the action expert network and output the predicted action sequence. In step four, enhance features t Input to Memory Conditioned Motor Expert Network: Cognitive Features t Provides high-level semantic information and perceives features. t It provides low-level visual details. The motion expert network outputs a continuous sequence of actions based on perceptual and cognitive features.

[0047] Step 5: When the capacity of the perception-cognition-reward memory bank reaches the preset limit, adjacent memory entries are merged based on a weighted index of feature similarity and reward similarity to implement memory consolidation; Step five specifically involves the following steps: In step S5, when the number of entries in the PCRMB reaches the preset capacity limit L, reward-based memory consolidation is implemented. This includes the following two sub-steps: S5.1: Redundant Memory Lookup. When an action is generated and new memory entries are continuously written to the PCRMB, if the current number of entries in the memory bank exceeds the maximum capacity L, a memory consolidation mechanism is triggered. A merging criterion is set. For every two adjacent memory entries, feature similarity and reward similarity are calculated, and the two similarities are weighted and merged to obtain the merging criterion. Specifically, two states are considered redundant information only when they are not only "visually and semantically similar" but also "have similar progress value to the task objective." The memory bank is traversed, and the pair of adjacent entries with the smallest criterion is selected for merging.

[0048] S5.2: Memory Consolidation and Update of PCRMB. A non-average merging strategy based on historical reward weights is used to avoid losing relatively high-value memory details. The calculated synthetic feature and synthetic reward are used as a new entry, along with the merged center timestamp, to replace two adjacent entries in the original memory bank, thus updating the PCRMB.

[0049] Furthermore, in step five, The step of using single-step dense reward as the advantage function and employing a policy optimization algorithm to perform gradient updates on the trainable parameters in the gated network specifically includes: using a valueless proximal policy optimization algorithm, after a preset number of time steps K, performing a gradient update on the trainable parameters in the gated network using a loss function constructed based on single-step dense reward, while keeping the parameters of the visual encoder, large language model, and action expert network frozen.

[0050] Furthermore, in step five, The step of merging adjacent memory entries based on a weighted index of feature similarity and reward similarity when the capacity of the perception-cognition-reward memory bank reaches its limit is as follows: When the number of entries in the perception-cognition-reward memory bank exceeds a preset capacity limit, a merging index is calculated between adjacent memory entries. This merging index is a weighted sum of the feature cosine distance and the reward Manhattan distance. The calculation method for the merging index is as follows:

[0051] in, , Enhance working memory for adjacent memory items; , For the corresponding historical rewards; , Preset weighting coefficients; Select the pair of adjacent entries with the smallest merging index, calculate the weighted average weight based on their historical rewards, merge the features and rewards, and generate a new memory entry.

[0052] Furthermore, Based on its historical rewards, a weighted average weight is calculated, and the features and rewards are merged to generate a new memory entry. Specifically: The historical rewards of two adjacent entries to be merged are converted into merge weights using a Softmax function with a temperature coefficient. The calculation method is as follows:

[0053]

[0054] in, Temperature coefficient; The calculation is performed after merging and memorizing; the calculation method is as follows:

[0055] The calculation method for the merged reward is as follows:

[0056] Replace the two existing adjacent entries with the merged memory, merged reward, and merged timestamp as new entries.

[0057] This invention proposes a dynamic memory enhancement system for a vision-language-action model, comprising: The working memory construction and retrieval module is used to acquire the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as a query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on time location encoding. This module observes sequences and natural language commands, outputting perceptual and cognitive tokens through a pre-trained visual encoder and a large language model, which are then concatenated into working memory. A PCRMB is designed to store historical entries (including historical memories and rewards) and to retrieve memories from the memory bank based on the current working memory.

[0058] The gated fusion module is used to input working memory and historical memory into a gated network with trainable parameters. The gated network calculates fusion weights and performs weighted fusion of current features and historical features to output enhanced working memory. This module includes a gated network with LoRA, which uses rewards to update fusion weights in real time via the PPO algorithm, and merges current and historical memories to output enhanced memory.

[0059] The progress assessment and reward generation module is used to use single-step dense reward as the advantage function, adopt a policy optimization algorithm to perform gradient updates on the trainable parameters in the gated network, and when the capacity of the perception-cognition-reward memory bank reaches the upper limit, perform reward-weighted merging of adjacent memory entries based on a weighted index of feature similarity and reward similarity. This module includes a progress evaluation network Φ, which outputs progress based on enhanced memory and calculates intensive rewards based on memory.

[0060] The online optimization and memory consolidation module is used to use single-step dense reward as the advantage function, and to perform gradient updates on the trainable parameters in the gated network using a policy optimization algorithm. When the capacity of the perception-cognition-reward memory bank reaches the upper limit, adjacent memory entries are merged based on a weighted index of feature similarity and reward similarity. This module is responsible for updating PCRMB, receiving reward signals, performing reward-based similarity calculations when the memory bank reaches its limit, and merging memory entries in a weighted manner according to the reward.

[0061] The action generation module is used to input enhanced working memory into the action expert network and output predicted action sequences.

[0062] This module receives enhanced working memory and outputs predicted action sequences through an action generation expert network.

[0063] This system incorporates the following innovations: First, it constructs a progress assessment and intensive reward module, introducing a progress assessment mechanism into the memory-enhanced VLA model. Relying on enhanced working memory features and language instruction embedding, it outputs single-step intensive rewards, providing a supervisory signal for real-time optimization of gating fusion weights. Second, it designs a dynamic gating fusion module. Addressing the issues of fixed gating parameters and the inability to adaptively adjust memory weights in the original model, it embeds a low-rank adapter into the gating network, combining intensive rewards to fine-tune a small number of parameters online, balancing dynamic weighting and real-time computation, completing multi-source memory fusion and updating gating weights. Third, it proposes a reward-aware memory consolidation module. It abandons the single cosine similarity mean merging method, integrates representation similarity and reward value distance to form a weighted evaluation criterion, aggregates memories through a reward-weighted averaging strategy, filters and retains high-value features, and achieves efficient compression and optimization of the memory bank.

[0064] This invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the above-described method.

[0065] The present invention will be further described in detail below with reference to specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments. Unless otherwise specified in the following embodiments, conventional conditions or existing test conditions were followed.

[0066] Example To verify the effectiveness of this dynamic memory enhancement method, integration tests were conducted based on an existing VLA model architecture. The basic VLA architecture consists of a visual encoder using a parallel structure of DINOv2 and SigLIP, a large language model using LLaMA-7B, and an action prediction head employing a policy network based on a diffusion model. The dynamic memory enhancement module of this invention is integrated as a plug-in onto this basic VLA architecture, enhancing its long-term memory capabilities through gating fusion and memory consolidation mechanisms.

[0067] Hyperparameter settings: Rank r=3 for the LoRA adapter in the gated network, learning rate lr = 1e-4, and merging metrics... =1.0, =5.0, Softmax temperature coefficient τ=0.05, update frequency K=8, PCRMB upper limit L=16.

[0068] Schedule estimator Input: Perception Token, Cognition Token, and Natural Language Command 1; Output: Estimated Progress Percentage. This experiment uses physical distance based on the robotic arm as progress. The preferred approach is to pre-train a VLM+MLP to evaluate progress, while a secondary approach is to use a large model. This experiment uses global pooling of the Perception Token and then concatenates it with the Cognition Token. The preferred approach is to use semantically guided weighted pooling.

[0069] Policy optimization algorithm: In this experiment, the post-training-based gated fusion part employs valueless proximal policy optimization and LoRA for fine-tuning, and treats the denoising loss of the diffusion process as a negative log-likelihood. Secondary options include online adjustment based on projective gradient descent and search based on evolutionary policies.

[0070] Action generation expert: This experiment uses a policy network based on a diffusion model. The preferred option is a policy network based on a flow model, and the second-best option is a policy network based on autoregressive sequence modeling.

[0071] Phase 1: (1) The system receives the current RGB image and instructions, and the feature extraction module outputs the current perception token and cognition token, and splices them together to form the current working memory.

[0072] (2) Using the current working memory as the Query, retrieve the historical information that best matches its semantics in PCRMB.

[0073] (3) The gated LoRA network allocates weights in real time, and integrates historical memory with current features into enhanced working memory.

[0074] (4) Based on enhanced working memory, the action generation expert generates a continuous action sequence through reverse denoising.

[0075] Phase Two: (1) The progress assessment module receives the enhanced working memory, calculates the task progress, and derives the intensive reward for that step. (2) The system caches memory entries and fine-tunes the gated LoRA parameters through Value-Free PPO every 8 steps.

[0076] (3) Monitor the capacity of PCRMB. If and only if the number of entries reaches the upper limit of 16, call the memory consolidation module and use the reward-weighted merging strategy to merge adjacent memory entries with extremely high similarity.

[0077] The benchmark simulation test uses the Bridge test suite in the SimplerEnv environment and sets four test tasks: taking a spoon from a towel, taking a carrot from a plate, stacking cubes, and taking an eggplant from a basket. Each task is repeated 24 times and the success rate of the task is calculated.

[0078] The baseline models used in this comparison were CogACT and π0. Experimental results show that, compared to the two baseline models, the MemTT-VLA model, equipped with the dynamic memory enhancement strategy proposed in this paper, improves the average success rate of the task by 14.6%, with a final success rate of 71.9%.

[0079] This dynamic memory enhancement method comprises three core algorithms: a progress evaluation and dense reward generation algorithm that utilizes a pre-trained progress evaluator to transform the fused enhanced memories into task progress values, providing optimization directions for subsequent weight updates and memory consolidation; a valueless proximal policy optimization algorithm that uses dense rewards as immediate feedback and employs lightweight trainable parameters for online fine-tuning of the gating network only during the testing period, enabling the model to enhance memory weights that contribute to the current task with extremely low latency; and a reward-aware memory consolidation algorithm that introduces a value dimension, combining "representation similarity" and "value similarity," where high-reward memory entries dominate the weights during merging, while simultaneously updating the perception-cognition-reward memory bank in real time.

[0080] This invention proposes a dynamic memory enhancement method and system for vision-language-action models. Addressing the problems of memory shortsightedness, memory contamination, and inability to dynamically adjust memory weights in existing VLA models for long-range tasks, this invention combines a dynamic memory enhancement mechanism with online adaptive optimization, specifically implemented through the following mechanism: Mechanism of combined memory generation: Based on the principles of cognitive science, observation and language instructions are compiled into low-level perceptual tokens and high-level cognitive tokens, which are combined to form working memory. A perceptual-cognitive-reward memory bank is designed to store historical memories and realize memory retrieval. Memory Enhancement Fusion Mechanism: Working memory composed of perceptual tokens and advanced cognitive tokens obtains historical memory through memory retrieval from the perceptual-cognitive memory bank. A gating fusion module is designed, a weight is trained, and working memory and historical memory are fused into enhanced working memory. In this way, the enhanced memory can maintain long-term memory. The calculated weight is updated in real time with rewards, which enhances the ability to face unknown environments. The progress-based reward calculation module inputs enhanced working memory into the progress evaluator to calculate progress and calculates real-time rewards based on progress. Since time encoding is added to the enhanced working memory, the progress evaluation has the ability to be aware of time sequence context. Perceptual-cognitive memory bank update mechanism: Define memory merging index, combine cosine similarity and reward difference to make merging judgment, and adopt reward-weighted averaging strategy to maintain the weight of high reward memory and weaken the weight of low reward memory.

[0081] This invention relates to the fields of embodied intelligence and robot learning technology, specifically disclosing a dynamic memory enhancement method and system for vision-language-action models. Addressing the memory shortsightedness and memory contamination problems of existing VLA models in long-range tasks, this invention proposes a dynamic memory enhancement framework, including: constructing a working memory containing perceptual and cognitive tokens, and retrieving historical information from a perceptual-cognitive-reward memory bank; adaptively fusing current observations and historical memories through a gating network with trainable parameters to output enhanced working memory; calculating task progress differences using a frozen progress estimator to generate single-step dense rewards; fine-tuning the gating network parameters in real time using a policy optimization algorithm based on these rewards, and executing a reward-weighted memory consolidation mechanism when the memory bank is full; and finally, generating action sequences based on the enhanced working memory using an action expert network. This invention, by introducing dynamic memory enhancement and online adaptive optimization, alleviates the memory contamination and distribution shift problems of existing VLA models in long-range tasks, achieving high-value memory retention and real-time adjustment in unknown dynamic environments.

[0082] The foregoing has provided a detailed description of the dynamic memory enhancement method, system, and readable storage medium for vision-language-action models proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A dynamic memory enhancement method for a vision-language-action model, characterized in that, Includes the following steps: Step 1: Obtain the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as the query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on the sinusoidal time step position encoding. Step 2: Input the working memory and historical memory into a gating network with trainable parameters, calculate the fusion weights through the gating network, and perform weighted fusion of the current features and historical features to output the enhanced working memory; Step 3: Input the enhanced working memory into the frozen progress evaluator to calculate the current task progress, generate a single-step dense reward based on the progress difference between adjacent time steps, and cache the memory tuple containing the single-step dense reward into the perception-cognition-reward memory bank. Step 4: Using single-step dense reward as the advantage function, the trainable parameters in the gated network are updated with gradients using a policy optimization algorithm. When the capacity of the perception-cognition-reward memory bank reaches its limit, adjacent memory entries are merged with reward weighting based on a weighted index of feature similarity and reward similarity. Step 5: Input the enhanced working memory into the action expert network and output the predicted action sequence.

2. The method according to claim 1, characterized in that, In step one, The process of acquiring the current observation data and natural language instructions, extracting perceptual tokens and cognitive tokens to construct working memory specifically involves: inputting the observation data into a pre-trained visual encoder to extract the original visual features, and then reducing the dimensionality of the data using a perceptual compression module to obtain the perceptual token; projecting the original visual features into the embedding space of a large language model, concatenating them with the word vectors of the natural language instructions, and then inputting them into the large language model to output the cognitive token; and concatenating the perceptual token and the cognitive token to form the working memory at the current moment. The process of using working memory as a query vector and retrieving historical memories from the perception-cognition-reward memory bank based on time position encoding specifically involves: adding the historical perception token and historical cognition token stored in the perception-cognition-reward memory bank to the sinusoidal time step position encoding as the key; using the perception token and cognition token in the current working memory as the query; calculating attention weights through a cross-attention mechanism; and retrieving the historical perception token, historical cognition token, and corresponding historical reward in a weighted manner.

3. The method according to claim 1, characterized in that, In step two, The calculation of fusion weights through the gating network specifically involves: concatenating the current perception token with the historical perception token to form a joint perception feature, and concatenating the current cognition token with the historical cognition token to form a joint cognition feature; inputting the joint perception feature and the joint cognition feature into the gating network respectively, and after joint calculation by the low-rank adaptive matrix and the frozen weight matrix, outputting the perception gating vector and the cognition gating vector through the Sigmoid activation function; The weighted fusion of current and historical features to output enhanced working memory specifically involves using the perceptual gating vector and cognitive gating vector to multiply the current and historical features element-wise and sum them to obtain the enhanced perceptual token and the enhanced cognitive token.

4. The method according to claim 1, characterized in that, The single-step intensive reward is the difference between the task progress at the current moment and the task progress at the previous moment.

5. The method according to claim 1, characterized in that, In step four, The step of using single-step dense reward as the advantage function and employing a policy optimization algorithm to perform gradient updates on the trainable parameters in the gated network specifically includes: using a valueless proximal policy optimization algorithm, after a preset number of time steps K, performing a gradient update on the trainable parameters in the gated network using a loss function constructed based on single-step dense reward, while keeping the parameters of the visual encoder, large language model, and action expert network frozen.

6. The method according to claim 1, characterized in that, In step four, The step of merging adjacent memory entries based on a weighted index of feature similarity and reward similarity when the capacity of the perception-cognition-reward memory bank reaches its limit is as follows: When the number of entries in the perception-cognition-reward memory bank exceeds a preset capacity limit, a merging index is calculated between adjacent memory entries. This merging index is a weighted sum of the feature cosine distance and the reward Manhattan distance. The calculation method for the merging index is as follows: in, , Enhance working memory for adjacent memory items; , For the corresponding historical rewards; , Preset weighting coefficients; Select the pair of adjacent entries with the smallest merging index, calculate the weighted average weight based on their historical rewards, merge the features and rewards, and generate a new memory entry.

7. The method according to claim 6, characterized in that, Based on its historical rewards, a weighted average weight is calculated, and the features and rewards are merged to generate a new memory entry. Specifically: The historical rewards of two adjacent entries to be merged are converted into merge weights using a Softmax function with a temperature coefficient. The calculation method is as follows: in, Temperature coefficient; The calculation method is as follows: (The calculation is then merged and memorized.) The calculation method for the merged reward is as follows: Replace the two existing adjacent entries with the merged memory, merged reward, and merged timestamp as new entries.

8. The method according to claim 1, characterized in that, The memory tuples stored in the perception-cognition-reward memory bank include enhanced working memory, corresponding single-step intensive rewards, and sinusoidal time-step position codes.

9. A dynamic memory enhancement system for a vision-language-action model, characterized in that, include: The working memory construction and retrieval module is used to acquire the observation data and natural language instructions at the current moment, extract the perception token and cognitive token to construct working memory, and use the working memory as a query vector to retrieve historical memories and corresponding rewards in the perception-cognition-reward memory bank based on time location encoding. The gated fusion module is used to input working memory and historical memory into a gated network with trainable parameters. The gated network calculates fusion weights and performs weighted fusion of current features and historical features to output enhanced working memory. The progress assessment and reward generation module is used to input the enhanced working memory into the frozen progress evaluator to calculate the current task progress, generate single-step dense rewards based on the progress difference between adjacent time steps, and cache the memory tuple containing the single-step dense rewards into the perception-cognition-reward memory bank. The online optimization and memory consolidation module is used to use single-step dense reward as the advantage function, and to perform gradient updates on the trainable parameters in the gated network using a policy optimization algorithm. When the capacity of the perception-cognition-reward memory bank reaches the upper limit, adjacent memory entries are merged based on a weighted index of feature similarity and reward similarity. The action generation module is used to input enhanced working memory into the action expert network and output predicted action sequences.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-8.

Images