Recovery method of vision-language-action model based on reinforcement learning

Abstract

The invention discloses a method for recovering a vision-language-action model based on reinforcement learning, and relates to the field of robot learning, the method provides a three-stage method for the deployment problem of the vision-language-action model, namely a VLA model, on resource-limited low-computing-power equipment, and the method comprises the steps that firstly, structured pruning is applied to the VLA model, and then the VLA model is constructed; then, a performance recovery stage is adopted, supervision fine tuning SFT and reinforcement learning RL are combined to recover the effectiveness of the model on downstream tasks, and finally quantitative adjustment is performed to further optimize the precision and efficiency of the model. According to the method, the VLA model can be normally operated on equipment with limited robot end resources, and the landing application of the VLA technology is promoted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot learning, and in particular to a method for restoring a vision-language-action model based on reinforcement learning. BACKGROUND

[0002] In recent years, the field of robot learning has made significant breakthroughs in the accuracy of policy execution and the generalization ability of tasks. Since the RT-2 model was proposed, vision-language-action (VLA) models have gradually attracted widespread attention. Such models are built on large-scale base models and exhibit strong generalization ability, pointing the way for the development of general-purpose robots capable of performing a variety of operational tasks. VLA models not only utilize the general knowledge contained in pre-trained vision-language models (VLMs), but also have the ability to understand language instructions, perceive visual environments, and generate reasonable actions.

[0003] Although VLA models exhibit the potential to achieve general-purpose robot capabilities, they mainly rely on large-scale VLMs, which face many challenges in actual deployment. Such models often contain a large number of parameters, resulting in high computational resource and memory overhead. Therefore, their efficient operation often depends on strong hardware support, which makes their deployment on resource-constrained robot platforms difficult. In addition, the inference speed of these models often cannot meet the needs of real-time applications, further limiting their applicability in dynamic and time-sensitive robot tasks.

[0004] To address the efficiency limitations of current VLA models, existing research has attempted to use various strategies to improve their computational efficiency while ensuring performance. One mainstream direction is to replace the backbone of the original model with a more lightweight and computationally efficient architecture, thereby reducing the model size and inference delay. In addition, using smaller-scale VLMs as base models also shows certain potential in balancing performance and resource overhead. Traditional model compression techniques, such as quantization, pruning, and knowledge distillation, are also widely used to reduce model size and computational burden. In terms of visual input, some research proposes to reduce redundant visual information while retaining key semantic content by optimizing visual token processing (such as more efficient token selection or adaptive token pruning). However, these methods still have deficiencies in terms of speed and memory efficiency, and do not fully consider the special adaptability of VLA models in robot control scenarios.

[0005] The present application aims at the deployment problem of the VLA model on low-computing-power devices with limited resources, and proposes a special recovery method for the compressed visual-language-action model based on reinforcement learning. Specifically, the method aims at the strict requirements of robots on computing resources, video memory and response speed in actual application, combines reinforcement learning technology, and optimizes the recovered VLA model after compression to ensure that it still has strong task execution ability under limited hardware conditions. This hierarchical method fully utilizes the advantages of various technologies and provides a feasible path for deploying the VLA model on resource-limited robot platforms. The method can be widely applied to various scenes that require running the VLA model on edge devices, such as service robots, inspection robots, security robots, etc. For example, in service robots, the method of the present application can enable robots to realize complex tasks such as natural language understanding, visual recognition and action planning in environments such as restaurants, hotels and shopping malls with lower hardware cost, effectively improving the interactive experience and task completion efficiency. Experimental results show that the proposed method can effectively compress the VLA model with minimal performance degradation, and even in some cases, it is superior to the original model, while greatly reducing the video memory occupancy and inference delay of the model.

[0006] Therefore, the skilled in the art is committed to developing a recovery method for a visual-language-action model based on reinforcement learning. SUMMARY

[0007] In view of the above defects of the prior art, the technical problem to be solved by the present application is how to locally deploy the VLA model under resource-limited conditions.

[0008] To achieve the above-mentioned purpose, the present application provides a recovery method for a visual-language-action model based on reinforcement learning, characterized in that the method proposes a three-stage method to solve the problem that the VLA model is difficult to run efficiently on low-computing-power devices with limited resources. First, structural pruning is applied to the VLA model, then a performance recovery stage is adopted, combining SFT and RL to recover the effectiveness of the model on downstream tasks, and finally, quantization adjustment is performed to further optimize the accuracy and efficiency of the model.

[0009] Further, the structural pruning includes the following steps: Step 1, constructing a structure dependency graph in a large language model (LLM); this coordinated pruning can avoid introducing inconsistencies or functional degradation in the model; Step 2, estimating the importance of each identified structure coupling group in the LLM according to its contribution score; applying these importance scores to the output of the model; Step 3, removing those groups with the lowest scores, assuming that they have the smallest contribution to task performance.

[0010] Further, in the step 1, according to the selected granularity, it can be applied at multiple levels, including at the MLP level, channel level or layer level; the selection of granularity reflects the trade-off between the degree of compression aggressiveness and model stability.

[0011] Further, in the implementation of the step 1, the LLM-Pruner pruning algorithm is used as an off-the-shelf structured pruning framework, specifically, pruning at the block level and using the Taylor importance criterion.

[0012] Further, in the implementation of the step 1, in order to maintain the representational ability and stability of the model, only the first and last layers of the decoder are retained, and only the intermediate layers are pruned.

[0013] Further, in the step 2, for a given group g, the method calculates its importance score I(g), and sorts it according to the importance score;

[0014] wherein, {g1, g2, …, g N} represents the set of all parameter groups, punred is the set of pruned parameter groups, retained is the set of retained parameter groups Further, the SFT re-trains the pruned VLA model using artificially labeled supervision, thereby enabling the model to adapt to its reduced architecture and restore most of its original functionality.

[0015] Further, the RL as a complementary optimization strategy, unlike supervised learning that relies solely on static labeled data, the RL can optimize long-term rewards by dynamically adjusting model parameters, making it particularly suitable for restoring delicate decision-making capabilities weakened by pruning and quantization.

[0016] Further, the structured pruning prunes 90% of the parameters of the original model, greatly reducing the model parameter quantity and computational complexity; the SFT is performed before the RL, so that the pruned model can restore most of the performance, greatly improving the exploration efficiency of the early stage of the RL.

[0017] Further, after the RL, 4-bit quantization is performed to achieve extreme memory compression.

[0018] This invention addresses the problems of traditional VLA models, such as large parameter size, high computational overhead, and difficulty in efficient operation on resource-constrained edge devices and embedded systems. It proposes a systematic three-stage compression and performance recovery scheme. This scheme, through structured pruning, supervised fine-tuning and reinforcement learning performance recovery, and further low-bit quantization, forms a complete, efficient, and scalable VLA model optimization method.

[0019] In terms of technical effectiveness, this invention can significantly reduce model memory usage and inference latency while maintaining or even improving the task performance and generalization ability of the original model. Specifically, through a high proportion of structured pruning, the model size and computational cost are greatly reduced. Coupled with reinforcement learning recovery techniques, the performance loss caused by compression is effectively compensated for, thus achieving the optimal balance between efficient compression and task execution capability. Furthermore, the use of 4-bit quantization technology enables model deployment and operation on some ultra-low-power devices, meeting the practical needs of various hardware environments.

[0020] In terms of application value, this invention possesses strong platform compatibility and flexible deployment capabilities, and can be widely applied to various robotic systems and edge computing devices requiring multimodal interaction and intelligent decision-making. For example, in scenarios such as service robots, industrial robots, intelligent security, smart healthcare, smart homes, and in-vehicle autonomous driving systems, this invention's technology can support complex human-computer interaction, multimodal perception, and autonomous decision-making tasks with limited computing resources, improving the system's practicality, intelligence level, and user experience.

[0021] In terms of industrialization prospects, this invention aligns with the current development trend of lightweight, edge-based, low-power, and high-performance AI models. It effectively reduces the hardware costs, maintenance costs, and energy consumption of AI systems, facilitating the shift of artificial intelligence technology from the cloud to the edge, and from high-performance computing platforms to more real-world terminal devices. Furthermore, the technical solution adopted in this invention possesses excellent standardization and modularity, facilitating seamless integration with existing artificial intelligence frameworks, robot operating systems, and IoT platforms, thus possessing the technological foundation and market potential for large-scale application.

[0022] In summary, this invention provides a practical and feasible solution for efficient compression and performance recovery of VLA models through technological innovation and system optimization. It has significant technical value, application value, and industrial transformation prospects, and is expected to promote the widespread implementation and development of multimodal AI technology in multiple fields such as robotics, edge computing, and the Internet of Things.

[0023] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0024] Figure 1 is a method structure diagram of a preferred embodiment of the present application; Figure 2 is a VLA structure change before and after pruning of a preferred embodiment of the present application; Figure 3 is a PPO reinforcement learning training flowchart of a preferred embodiment of the present application. DETAILED DESCRIPTION

[0025] The following reference to the drawings accompanying the specification introduces a plurality of preferred embodiments of the present application, so that the technical content thereof is more clear and easy to understand. The present application can be embodied in many different forms of embodiments, and the protection scope of the present application is not limited to the embodiments mentioned herein.

[0026] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0027] The present method proposes a three-stage method, as shown in Figure 1 : (1) applying structured pruning to the VLA model; (2) using a performance recovery stage, combining the SFT method of the present method and the RL method of the present method, to recover the effectiveness of the model on downstream tasks; (3) performing quantization adjustment to further optimize the accuracy and efficiency of the model

[0028] 1.1 Structured pruning of the VLA of the present method

[0029] In the VLA of the present method, most of the parameters are concentrated in the LLM component, so it is a natural target for compression to reduce the overall computational complexity. The present method particularly uses a structured pruning technique, rather than an unstructured pruning, because structured pruning has inherent hardware efficiency advantages, thereby being able to achieve more significant and practical acceleration when deployed.

[0030] The first step of applying structured pruning to the VLA usually involves constructing a structure dependency graph in the LLM. This coordinated pruning can avoid introducing inconsistencies or functional degradation in the model. According to the selected granularity, structured pruning can be applied at multiple levels, such as at the level of the MLP of the present method, at the channel level, or at the layer level. The choice of granularity reflects the trade-off between the degree of compression aggressiveness and the stability of the model.

[0031] Next, the method estimates the importance of each identified structural coupling group in the LLM according to its contribution degree, and applies these importance scores to the output of the model. Specifically, for a given group g, the method computes its importance score I(g), and ranks it according to the importance score. Then, the method removes those groups with the lowest scores, assuming that they contribute the least to the task performance.

[0032]

[0033] where, {g1, g2, …, g N} represents the set of all parameter groups, punred is the set of pruned parameter groups, retained is the set of remaining parameter groups In the implementation of the method, the method uses the method LLM-Pruner as an off-the-shelf structured pruning framework. Specifically, the method prunes at the block level and uses the Taylor importance criterion. To maintain the representational capacity and stability of the model, the method keeps the first and last layers of the decoder and only prunes the intermediate layers. According to the empirical observations in the early experiments of the method, the method uses an aggressive 90% overall pruning ratio, aiming to significantly reduce the model size.

[0034] Figure 2 shows the architecture transition of the method VLA before and after structured pruning, focusing on the attention and MLP layers in the LLM component. Notably, the intermediate dimension of the linear layers in the self-attention and feed-forward submodules is significantly reduced, leading to a substantial reduction in overall parameter quantity and computational cost. Importantly, the input and output dimensions of each decoder layer remain unchanged, maintaining structural compatibility and enabling seamless integration with downstream modules. This design ensures that the pruned model retains its functional interface while achieving significant compression. The subsequent performance recovery stage of the method shows that the performance of the pruned method VLA can be effectively recovered, even under such an aggressive compression ratio, and even exceeds the performance of the original model.

[0035] 1.2 Performance recovery of the method based on SFT and RL

[0036] Compared with unstructured pruning, structured pruning removes individual weights in a fine-selective manner, which usually has a greater impact on model performance. This performance degradation is particularly evident at high pruning ratios, such as the 90% global pruning rate adopted in the setting of the proposed method supervised fine-tuning. To mitigate this loss and restore the model's task execution ability, the proposed method naturally employs the proposed SFT on task-specific data. By using artificially labeled supervised retraining of the pruned VLA model, SFT enables the model to adapt to its reduced architecture and restore most of its original functionality.

[0037] However, the proposed method observes that the performance of the pruned VLA model cannot be fully restored using SFT alone. To address this issue, the proposed method turns to RL as a complementary optimization strategy. Unlike supervised learning, which relies solely on static labeled data, RL can optimize long-term rewards by dynamically adjusting model parameters, making it particularly suitable for restoring the delicate decision-making ability weakened by pruning and quantization.

[0038] The proposed method employs Proximal Policy Optimization (PPO) as the reinforcement learning algorithm due to its stability and efficiency in high-dimensional action spaces, and its training process flowchart is shown in Figure 3. The objective of PPO is defined as follows:

[0039] where the proposed represents the probability ratio between the new policy and the old policy, where and represent the action and state at time t, respectively. represents the new policy with as the parameter. represents the previous policy before updating. is the estimated advantage function. is a hyperparameter that controls the clipping range.

[0040] In addition, since the pruned VLA model of the proposed method still retains a high task success rate after a significant compression and has obtained a good initialization through SFT (supervised fine-tuning), it provides a strong starting point for PPO-based reinforcement learning, making it naturally suitable for a lightweight and efficient RLFT (reinforcement learning fine-tuning) scheme.

[0041] 1.3 Further 4-bit quantization of the proposed method

[0042] After supervised fine-tuning (SFT) and reinforcement learning (RL), the pruned VLA has reached or even surpassed the level of the original VLA in task performance. On this solid foundation, the method further explores 4-bit quantization to achieve extreme memory compression. This additional step significantly reduces the memory usage of the model, enabling it to be successfully deployed in resource-constrained environments. By integrating structured pruning, performance recovery, and quantization into a unified processing flow, the method verifies that while meeting the strict requirements of efficiency in the real world, the model can still maintain high efficiency and practicality. It is worth noting that applying quantization to a VLA model that has been pruned by 90% can further reduce memory consumption, but it will have some impact on other performance indicators. Therefore, if the target deployment device has sufficient memory, quantization is not a necessary step, so the method calls it optional quantization.

[0043] In summary, the method integrates structured pruning, performance recovery, and quantization into a unified compression framework for VLA. By focusing on structured pruning of large language model (LLM) parts, the method significantly reduces model size and inference cost without affecting hardware efficiency. The two-stage performance recovery strategy composed of SFT and RL is particularly suitable for restoring the model's task execution ability at high pruning ratios. Finally, 4-bit quantization is used to further compress memory usage, enabling the model to be deployed on resource-constrained robot platforms. Overall, these technical means show that the VLA model can be significantly compressed while maintaining task performance, paving the way for efficient and scalable deployment in real-world environments.

Claims

1. A method for recovering a vision-language-action model based on reinforcement learning, characterized in that, To solve the problem that the VLA model cannot run efficiently on low-power devices with limited resources, a three-stage method is proposed. First, structural pruning is applied to the VLA model. Then, a performance recovery stage is used to recover the effectiveness of the model on downstream tasks by combining SFT and RL. Finally, quantization adjustment is performed to further optimize the accuracy and efficiency of the model. 2.The method of claim 1, wherein, The structural pruning includes the following steps: Step 1: Construct a structural dependency graph in a large language model (LLM). This coordinated pruning can avoid introducing inconsistencies or functional degradation in the model. Step 2: Estimate the importance of each identified structural coupling group in the LLM based on its contribution. Apply these importance scores to the output of the model. Step 3: Remove those groups with the lowest scores, assuming they have the smallest contribution to task performance. 3.The method of claim 2, wherein, In Step 1, the selected granularity can be applied at multiple levels, including MLP level, channel level, or layer level. The choice of granularity reflects the trade-off between the degree of compression aggressiveness and model stability. 4.The method of claim 3, wherein, In the implementation of Step 1, the LLM-Pruner pruning algorithm is used as an off-the-shelf structured pruning framework. Specifically, pruning is performed at the block level, and the Taylor importance criterion is used. 5.The method of claim 4, wherein, In the implementation of Step 1, to maintain the representational capacity and stability of the model, only the first and last layers of the decoder are retained, and only the intermediate layers are pruned. 6.The method of claim 5, wherein, In said step 2, for a given group g, the method computes its importance score I (g), and ranks them according to their importance score; wherein, = {g1, g2,..., g N} represents a set of all parameter groups, punred is a set of pruned parameter groups, retained is a set of remaining parameter groups. 7.The method of claim 6, wherein, The SFT re-trains the pruned VLA model using artificially annotated supervision, enabling the model to adapt to its reduced architecture and restore most of its original functionality. 8.The method of claim 7, wherein, The RL serves as a complementary optimization strategy. Unlike supervised learning that relies solely on static labeled data, the RL can optimize long-term rewards by dynamically adjusting model parameters, making it particularly suitable for restoring the nuanced decision-making ability weakened by pruning and quantization. 9.The method of claim 8, wherein, The structural pruning removes 90% of the parameters of the original model, greatly reducing the model parameter quantity and computational complexity. The SFT is performed before the RL, allowing the pruned model to recover most of its performance and significantly improving the exploration efficiency in the early stage of the RL. 10.The method of claim 9, wherein, After the RL, 4-bit quantization is performed to achieve extreme memory compression.

Images