VLA model optimization method and device based on iterative reinforcement learning, equipment and medium

Abstract

The invention discloses a VLA model optimization method and device based on iteration reinforcement learning, equipment and a medium, and relates to the field of artificial intelligence, so that a robot can accurately execute related tasks in financial, insurance and medical scenes, and the method comprises the steps: carrying out the supervised learning pre-training of a preset initial VLA model, obtaining a pre-trained VLA model, and carrying out the optimization of the pre-trained VLA model; the pre-trained VLA model comprises an action decoder; all parameters except the action decoder in the pre-trained VLA model are frozen, online reinforcement learning is carried out on the action decoder, the parameters of the action decoder are updated, and a first optimized VLA model is obtained; and sending the first optimized VLA model to a server, unfreezing all parameters of the first optimized VLA model by the server, performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining a second optimized VLA model. According to the method, the instability of the VLA model in the reinforcement learning optimization process can be effectively overcome, and the computing resource and real-time bottleneck caused by full-parameter fine tuning is broken through.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a VLA model optimization method, device, equipment and medium based on iterative reinforcement learning. Background Art

[0002] The breadth and depth of intelligent robotic applications are expanding in sectors like finance, insurance, and healthcare, where operational precision and reliability are critical. Ensuring that robots can accurately understand complex instructions and execute tasks safely and efficiently has become a core concern for both industry and academia.

[0003] Vision-Language-Action (VLA) models, with their powerful cross-modal information understanding and generation capabilities, have shown significant potential in robotic control. These models are typically based on large-scale pre-training and, through supervised fine-tuning (SFT) on high-quality expert-led datasets, learn to directly map visual observations and natural language commands into low-level robot control signals (such as joint angles and end-user pose). This approach simplifies the control process to a certain extent, reducing reliance on traditional complex hierarchical control architectures.

[0004] However, when further optimizing the VLA model to adapt to the dynamic and complex tasks in real physical environments, existing technologies face two key challenges:

[0005] Insufficient reinforcement learning optimization stability: When attempting to apply online reinforcement learning (RL) strategies to optimize behavioral policies end-to-end for large-scale VLA models, training instability is a common problem. This instability manifests as difficulty in policy convergence and even significant degradation during learning. The root cause lies in the inherent conflict between the complex parameter space of large models and the exploration-exploitation mechanism of reinforcement learning. This leads to noisy gradient estimates and difficulty in precisely controlling the direction of policy updates, ultimately hindering the effective improvement of the model's robustness and generalization capabilities in real-world environments.

[0006] Computing resource bottleneck for full parameter fine-tuning: Large VLA models contain a vast number of parameters, and full parameter fine-tuning requires enormous computing resources. This often exceeds the capacity of conventional local computing devices. While relying on remote high-performance computing servers (such as cloud computing platforms) can provide computing power, the resulting network communication latency significantly reduces the responsiveness of robot control. In scenarios requiring high real-time performance, such as financial transactions, precision medical procedures, or complex insurance inspections, this latency is unacceptable and severely restricts the practical deployment of the model.

[0007] Therefore, how to effectively overcome the instability of the VLA model during the reinforcement learning optimization process, break through the computing resources and real-time bottlenecks brought about by full parameter fine-tuning, and develop an efficient, stable and deployable model optimization method is a core technical problem that needs to be solved urgently to improve the task execution capabilities of intelligent robots in the above-mentioned key areas. Summary of the Invention

[0008] The embodiments of the present invention provide a VLA model optimization method, apparatus, device, and medium based on iterative reinforcement learning, aiming to effectively overcome the instability of the VLA model during the reinforcement learning optimization process, break through the computing resource and real-time bottlenecks brought about by full parameter fine-tuning, and provide an efficient, stable, and deployable model optimization method.

[0009] In a first aspect, an embodiment of the present invention provides a VLA model optimization method based on iterative reinforcement learning, which includes:

[0010] Performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder;

[0011] Freezing all parameters of the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model;

[0012] The first optimized VLA model is sent to the server, which unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

[0013] A further technical solution is that the action decoder is subjected to online reinforcement learning, and the parameters of the action decoder are updated to obtain a first optimized VLA model, including:

[0014] Collecting environmental images and instruction information, and generating a fusion feature vector based on the environmental images and instruction information;

[0015] inputting the fusion feature vector into the pre-trained VLA model and receiving robot trajectory data output by the action decoder;

[0016] determining whether the robot trajectory data is successful trajectory data;

[0017] if the robot trajectory data is successful trajectory data, updating parameters of the action decoder based on the robot trajectory data using a preset reinforcement learning algorithm.

[0018] A further technical solution is that the fusion feature vector is generated based on the environment image and instruction information, and includes:

[0019] encoding the environment image into an image feature vector by a preset image encoder;

[0020] encoding the instruction information into an instruction feature vector by a preset instruction encoder;

[0021] generating a fusion feature vector according to the image feature vector and the instruction feature vector.

[0022] A further technical solution is that the determination of whether the robot trajectory data is successful trajectory data includes:

[0023] determining whether the robot trajectory data meets a trajectory standard corresponding to the instruction information;

[0024] if the trajectory data meets the trajectory standard corresponding to the instruction information, determining that the robot trajectory data is successful trajectory data;

[0025] if the trajectory data does not meet the trajectory standard corresponding to the instruction information, determining that the robot trajectory data is not successful trajectory data.

[0026] A further technical solution is that the method further includes:

[0027] if the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data;

[0028] storing the successful sample data into a successful sample set.

[0029] A further technical solution is that the supervised learning training of the first optimized VLA model, the updating of the parameters of the first optimized VLA model, and the obtaining of a second optimized VLA model include:

[0030] obtaining an expert sample set and the successful sample set, generating a training data set based on the expert sample set and the successful sample set;

[0031] The first optimized VLA model is trained through supervised learning using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

[0032] A further technical solution is that the first optimized VLA model is subjected to supervised learning training using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model, including:

[0033] Collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining an output result of the first optimized VLA model;

[0034] Determining a loss amount based on the output result and a correct result corresponding to the training sample;

[0035] Parameters of the first optimized VLA model are updated according to the loss amount.

[0036] In a second aspect, an embodiment of the present invention further provides a VLA model optimization device based on iterative reinforcement learning, which includes a unit for executing the above method.

[0037] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0039] The embodiment of the present application provides a VLA model optimization method and device based on iterative reinforcement learning, equipment and medium. Wherein, the method comprises: performing supervised learning pre-training on a preset initial VLA model to obtain a pre-training VLA model, wherein the pre-training VLA model comprises an action decoder; freezing all parameters of the pre-training VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model; and sending the first optimized VLA model to a server, unfreezing all parameters of the first optimized VLA model by the server, and performing supervised learning training on the first optimized VLA model to update the parameters of the first optimized VLA model and obtain a second optimized VLA model. Through the stage-by-stage iterative optimization strategy, the initial VLA model is first pre-trained by supervised learning to form a pre-training VLA model with basic task understanding ability; then all parameters of the pre-training VLA model except the action decoder are frozen for online reinforcement learning, so that the pre-training VLA model retains the general semantic understanding ability while only optimizing the environmental adaptability of the action decoder. This local parameter updating mechanism significantly reduces the computational load and training instability in the reinforcement learning process, and avoids the gradient disorder risk caused by fine-tuning of all parameters of the VLA model. Finally, the server unfreezes all parameters of the VLA model for supervised learning, and uses remote computing power to realize global fine-tuning. This design creatively balances the model optimization efficiency and performance: the local stage focuses on real-time environmental adaptation of the action strategy (ensures control frequency), and the server stage completes knowledge integration (improves precision), which fundamentally solves the collapse risk and computing resource bottleneck problem of full-parameter online reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0041] Figure 1 The flowchart of the VLA model optimization method based on iterative reinforcement learning provided by the embodiment of the present application is shown.

[0042] Figure 2 The schematic block diagram of the VLA model optimization device based on iterative reinforcement learning provided by the embodiment of the present application is shown.

[0043] Figure 3 The schematic block diagram of the computer equipment provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0046] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0047] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0049] See also Figure 1 To improve the intelligent application of intelligent robots in financial, insurance, and medical application scenarios, an embodiment of the present invention provides a VLA model optimization method based on iterative reinforcement learning, which includes the following steps:

[0050] S1, performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder.

[0051] In specific implementations, the initial VLA model refers to a VLA model that has not been pre-trained. This model is pre-trained through supervised learning to produce a pre-trained VLA model. The pre-trained VLA model includes an action decoder that outputs robot trajectory data. First, the initial VLA model undergoes supervised learning pre-training to produce a pre-trained VLA model with basic task understanding capabilities.

[0052] Specifically, the process of supervised learning pre-training includes: processing the input environmental visual image and natural language instructions through the image encoder and instruction encoder respectively to generate corresponding image feature vectors and instruction feature vectors; then the two types of feature vectors are spliced ​​to form a fusion vector, and the fusion vector is input into the initial VLA model for cross-modal semantic parsing. The initial VLA model outputs an action feature vector based on the fusion feature, and the action feature vector is further passed to the action decoder to generate the predicted value of the robot action. In order to optimize the model parameters, the predicted action is compared dimension by dimension with the standard action marked in the expert dataset, and the mean squared error (MSE) is calculated based on the difference. Finally, the gradient descent is performed through the back-propagation algorithm, and the trainable parameters of the initial VLA model are updated synchronously, so that the initial VLA model gradually converges to a state that can accurately map the visual-language input to the action output, that is, the pre-trained VLA model is obtained.

[0053] S2, freezing all parameters in the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model.

[0054] In practice, all parameters of the pre-trained VLA model, except for the action decoder, are frozen for online reinforcement learning. This allows the model to retain its general semantic understanding capabilities while optimizing only the action generation module for environmental adaptability. This local parameter update mechanism significantly reduces the computational load and training instability during reinforcement learning. By updating only the action decoder parameters, the number of trainable parameters is significantly reduced, thereby reducing gradient noise and policy update instability in reinforcement learning, and avoiding the risk of gradient disorder during full parameter fine-tuning.

[0055] For example, in some preferred embodiments, the above step of "performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model" specifically includes the following steps: collecting environmental images and instruction information, and generating a fused feature vector based on the environmental images and instruction information; inputting the fused feature vector into the pre-trained VLA model, and receiving the robot trajectory data output by the action decoder; determining whether the robot trajectory data is successful trajectory data; if the robot trajectory data is successful trajectory data, based on the robot trajectory data, using a preset reinforcement learning algorithm to update the parameters of the action decoder.

[0056] In a specific implementation, the environment image can be captured by the robot's camera, which is not specifically limited by the present invention; the command information can be input by the user, which is not specifically limited by the present invention. A fused feature vector is generated by combining the environment image and the command information to drive action decoding, and the validity of the robot's trajectory data is verified based on preset rules. This design directly links the reinforcement learning reward signal to the essential task goal (command completion), establishing a closed-loop self-verification mechanism.

[0057] Furthermore, the action decoder's parameters are only updated when a trajectory is deemed successful, ensuring that the action decoder's optimization direction always converges toward the feasible solution space. This effectively suppresses random behavioral interference during reinforcement learning exploration, avoids the risk of policy degradation caused by ineffective exploration, and significantly improves online learning stability and sample utilization efficiency.

[0058] Furthermore, in some preferred embodiments, the above step of "generating a fused feature vector based on the environmental image and the instruction information" specifically includes the following steps: encoding the environmental image into an image feature vector through a preset image encoder; encoding the instruction information into an instruction feature vector through a preset instruction encoder; and generating a fused feature vector based on the image feature vector and the instruction feature vector.

[0059] In a specific implementation, the image encoder and the instruction encoder may be specifically a feature extraction module, which is not specifically limited in the present invention. The fused feature vector may be formed by concatenating the image feature vector and the instruction feature vector.

[0060] In the present invention, the visual and language modalities are processed separately by independent encoders (the image encoder extracts spatial features, and the instruction encoder parses semantic intent), and then fused to generate a fused feature vector, thereby achieving decoupling and alignment of cross-modal information.

[0061] Furthermore, in some preferred embodiments, the above step of "determining whether the robot trajectory data is successful trajectory data" specifically includes the following steps: determining whether the robot trajectory data meets the trajectory standard corresponding to the instruction information; if the trajectory data meets the trajectory standard corresponding to the instruction information, determining that the robot trajectory data is successful trajectory data; if the trajectory data does not meet the trajectory standard corresponding to the instruction information, determining that the robot trajectory data is not successful trajectory data.

[0062] In a specific implementation, the trajectory standard is pre-set according to the task objective of the instruction information. Specifically, the trajectory standard can be set by the user, and the present invention does not specifically limit this. For example, the instruction information can be specifically to pick up a water cup on the table, then the trajectory standard can be that the robot uses its mechanical hand to hold the water cup and lift the water cup off the table.

[0063] In the present invention, by converting abstract success judgments into specific trajectory standards bound to instructions, a quantifiable task completion evaluation system is established, thereby greatly improving the efficiency and accuracy of determining whether the robot trajectory is successful.

[0064] In some preferred embodiments, the method further includes: if the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data; and storing the successful sample data in a successful sample set.

[0065] In a specific implementation, a piece of successful sample data is generated based on the successful trajectory data and the corresponding environment image, and the successful sample data is stored in a successful sample set.

[0066] In this invention, a sample dataset is constructed simultaneously with the generation of successful trajectory data, enabling the accumulation of knowledge from online experience. Converting successful cases obtained through reinforcement learning exploration into supervised learning samples essentially solidifies the results of strategy exploration in a dynamic environment into static knowledge. This experience replay mechanism not only expands the diversity of training data (covering actual environmental noise and boundary conditions), but also builds a knowledge transfer bridge between reinforcement learning and supervised learning, providing high-quality real-world scenario samples for subsequent global fine-tuning.

[0067] S3: Send the first optimized VLA model to the server. The server unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

[0068] In a specific implementation, the first optimized VLA model is sent to the server, which unfreezes all parameters of the first optimized VLA model and performs supervised learning training on the first optimized VLA model. The parameters of the first optimized VLA model are updated to obtain the second optimized VLA model. By unfreezing all parameters of the first optimized VLA model on the server for supervised learning and leveraging remote computing power for global fine-tuning, the accuracy of the second optimized VLA model is greatly improved.

[0069] For example, in some preferred embodiments, the above step of "performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model" specifically includes the following steps: obtaining an expert sample set and the successful sample set, and generating a training data set based on the expert sample set and the successful sample set; performing supervised learning training on the first optimized VLA model through the training data set, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model.

[0070] In a specific implementation, the expert sample set can be input by a user, which is not specifically limited by the present invention. The expert sample set includes multiple high-accuracy samples produced by experts in the field. The successful sample set can be uploaded by the terminal to the server. The expert sample set and the successful sample set are combined to obtain a training dataset. Furthermore, the first optimized VLA model is trained through supervised learning using the training dataset, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

[0071] In this invention, a training dataset is obtained by fusing an expert sample set constructed by experts in the field with a successful sample set constructed by successful samples generated by reinforcement learning. This enables dual knowledge distillation: the expert sample set ensures that basic task capabilities are not degraded, and the successful sample set injects new knowledge for environmental adaptation. This strategy enables global supervised learning on the server side to simultaneously absorb human expert experience (safety) and the results of machine autonomous exploration (environmental robustness), significantly improving the model's generalization ability in complex physical scenarios. In particular, the real-world positive and negative feedback provided by the successful sample set can effectively improve the robot's environmental adaptability.

[0072] In some preferred embodiments, the above step of "performing supervised learning training on the first optimized VLA model through the training data set, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model" specifically includes the following steps: collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining the output results of the first optimized VLA model; determining the loss amount based on the correct results corresponding to the output results and the training samples; and updating the parameters of the first optimized VLA model according to the loss amount.

[0073] In a specific implementation, the loss between the output result and the correct result can be specifically the mean squared error (MSE). Based on the mean squared error, the parameters of the first optimized VLA model are updated through the back propagation algorithm to obtain the second optimized VLA model.

[0074] In this paper, a standard supervised learning framework is used to perform global fine-tuning, with parameter updates driven by loss calculation. A loss feedback mechanism ensures that the model output is aligned with the distribution of the training dataset (experts + successful samples), ultimately achieving end-to-end consistency across the perception-decision-execution chain, fundamentally addressing the inter-module adaptation bias that can arise from phased optimization.

[0075] Furthermore, after obtaining the second optimized VLA model, the second optimized VLA model can be sent to the terminal, and reinforcement learning training can be performed at the terminal, that is, steps S2-S3 are performed alternately, so that the model has better performance.

[0076] An embodiment of the present invention proposes a VLA model optimization method based on iterative reinforcement learning, including: performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder; freezing all parameters in the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model; sending the first optimized VLA model to a server, which unfreezes all parameters of the first optimized VLA model and performs supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining a second optimized VLA model. The present invention adopts a phased iterative optimization strategy, first performing supervised learning pre-training on the initial VLA model to form a pre-trained VLA model with basic task understanding capabilities; then freezing all parameters of the pre-trained VLA model except the action decoder for online reinforcement learning, so that the pre-trained VLA model retains general semantic understanding capabilities while only optimizing the environmental adaptability of the action decoder. This local parameter update mechanism significantly reduces the computational load and training instability during reinforcement learning, avoiding the risk of gradient disorder caused by fine-tuning all parameters of the VLA model. Finally, all parameters of the VLA model are unfrozen on the server for supervised learning, and global fine-tuning is achieved using remote computing power. This design creatively balances model optimization efficiency and performance: the local stage focuses on real-time environmental adaptation of the action strategy (ensuring control frequency), while the server stage completes knowledge integration (improving accuracy), fundamentally addressing the crash risk and computing resource bottleneck issues of full-parameter online reinforcement learning.

[0077] This embodiment of the present invention provides a VLA model optimization method based on iterative reinforcement learning. This method significantly improves the VLA model's task understanding accuracy and action execution reliability in dynamic physical scenarios through a phased parameter optimization mechanism and a cross-environment knowledge transfer strategy. This optimized model can efficiently drive robots to perform complex tasks in applications such as finance, insurance, and healthcare, where operational safety, standardization, and real-time performance are critical. These tasks include, but are not limited to:

[0078] Application scenarios in the financial field, such as:

[0079] Physical document processing in securities business: Bank robots need to accurately locate target documents based on customer voice commands (such as "prioritize expedited remittance orders") and perform operations such as stamping, sorting, and archiving to avoid the risk of mistakenly taking documents with adjacent numbers.

[0080] Vault asset inventory: In a security monitoring environment, the robot autonomously plans an obstacle avoidance path according to risk control instructions (such as "scan precious metals on the B layer of the third safe") to complete the physical inventory and account verification.

[0081] Emergency intervention in the trading hall: When the system detects abnormal trading behavior, the robot receives instructions (such as "immediately suspend terminal A003 device") and quickly moves to the target terminal to perform a physical power-off operation.

[0082] Application scenarios in the insurance field, such as:

[0083] Accident site investigation and damage assessment: The investigation robot autonomously adjusts the gimbal angle according to background instructions (such as "take a close-up of the vehicle chassis damage"), completes multi-angle image acquisition in complex terrain, and avoids omissions from manual investigation perspectives.

[0084] Intelligent management of insurance policy files: Based on audit requirements (such as "retrieve the 2024 health insurance claim denial case"), the robot accurately extracts the target file box from the dense file rack to avoid colliding with adjacent files and causing disorder.

[0085] Inspection of disaster prevention supplies: Identify shelf levels according to flood prevention instructions (such as "check the reserve of life jackets in warehouse No. 3"), unpack and verify the quantity of supplies and report any damage.

[0086] Application scenarios in the medical field, such as:

[0087] Precise delivery of sterile instruments: The operating room robot delivers instruments along a sterile path within a limited operating space according to the surgeon's instructions (such as "deliver the microneedle holder to area 2"), avoiding contaminated areas and maintaining a safe operating distance.

[0088] Transfer of high-risk samples: When the laboratory robot executes instructions (such as "transfer HIV serum samples to the P3 laboratory"), it automatically selects an anti-shock path and monitors the sealing status of the transport box in real time.

[0089] Emergency drug delivery: After receiving instructions from the pharmacy (such as "urgently deliver epinephrine to bed 3 in the emergency room"), the robot dynamically plans the shortest path and identifies the location of the bed, ensuring that emergency drugs are delivered in seconds.

[0090] Specific application cases include:

[0091] Insurance claims scenario: The optimized VLA model drives a survey robot to execute the command to "photograph the deformed area of ​​the vehicle's A-pillar" at the scene of a rainstorm. It successfully integrates visual recognition (locating the A-pillar), language understanding (eliminating interference from the B-pillar), and motion control (stable focus in the rain) to accurately complete the task.

[0092] Operating room scenario: In a simulated prostatectomy, when the model-controlled instrument delivery robot responded to the instruction "translate 3 cm to the left to avoid the nerve bundle," the motion trajectory error was controlled within 0.5 mm, avoiding the risk of accidental touch caused by language ambiguity in traditional methods.

[0093] Financial security scenario: In response to the command to "lock vault channel 7," the model-driven security robot completed the triple operations of physical access locking, channel barrier deployment, and intrusion detection system activation within 40 seconds, meeting the atomicity requirements of financial security protocols for multiple action sequences.

[0094] This method optimizes action strategies through local reinforcement learning to ensure real-time performance, global fine-tunes the server to improve accuracy, and aligns cross-modal features to avoid misoperation. It systematically solves the problem of collaborative optimization of environmental dynamics, operational standardization, and immediate response in complex domain tasks.

[0095] See also Figure 2 , Figure 2 : is a schematic block diagram of a VLA model optimization device 20 based on iterative reinforcement learning provided by an embodiment of the present invention. Corresponding to the above VLA model optimization method based on iterative reinforcement learning, the present invention also provides a VLA model optimization device 20 based on iterative reinforcement learning. The VLA model optimization device 20 based on iterative reinforcement learning includes a unit for executing the above VLA model optimization method based on iterative reinforcement learning. The VLA model optimization device 20 based on iterative reinforcement learning can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, the VLA model optimization device 20 based on iterative reinforcement learning includes:

[0096] A pre-training unit 21 is configured to perform supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder;

[0097] A reinforcement learning unit 22 is configured to freeze all parameters of the pre-trained VLA model except the action decoder, perform online reinforcement learning on the action decoder, update the parameters of the action decoder, and obtain a first optimized VLA model;

[0098] The supervised learning unit 23 is used to send the first optimized VLA model to the server, and the server unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

[0099] In some preferred embodiments, performing online reinforcement learning on the action decoder to update parameters of the action decoder to obtain a first optimized VLA model includes:

[0100] Collecting environmental images and instruction information, and generating a fusion feature vector based on the environmental images and instruction information;

[0101] Inputting the fused feature vector into the pre-trained VLA model and receiving the robot trajectory data output by the action decoder;

[0102] Determining whether the robot trajectory data is successful trajectory data;

[0103] If the robot trajectory data is successful trajectory data, a preset reinforcement learning algorithm is used based on the robot trajectory data to update the parameters of the action decoder.

[0104] In some preferred embodiments, generating a fused feature vector based on the environment image and the instruction information includes:

[0105] Encoding the environment image into an image feature vector by a preset image encoder;

[0106] Encoding the instruction information into an instruction feature vector by a preset instruction encoder;

[0107] A fusion feature vector is generated according to the image feature vector and the instruction feature vector.

[0108] In some preferred embodiments, determining whether the robot trajectory data is successful trajectory data includes:

[0109] Determining whether the robot trajectory data meets the trajectory standard corresponding to the instruction information;

[0110] If the trajectory data meets the trajectory standard corresponding to the instruction information, the robot trajectory data is determined to be successful trajectory data;

[0111] If the trajectory data does not meet the trajectory standard corresponding to the instruction information, it is determined that the robot trajectory data is not successful trajectory data.

[0112] In some preferred embodiments, the method further comprises:

[0113] If the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data;

[0114] The successful sample data is stored in a successful sample set.

[0115] In some preferred embodiments, performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model include:

[0116] Acquire an expert sample set and the successful sample set, and generate a training data set based on the expert sample set and the successful sample set;

[0117] The first optimized VLA model is trained through supervised learning using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

[0118] In some preferred embodiments, performing supervised learning training on the first optimized VLA model using the training data set, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model includes:

[0119] Collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining an output result of the first optimized VLA model;

[0120] Determining a loss amount based on the output result and a correct result corresponding to the training sample;

[0121] Parameters of the first optimized VLA model are updated according to the loss amount.

[0122] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned VLA model optimization device 20 based on iterative reinforcement learning and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and conciseness of the description, it will not be repeated here.

[0123] The VLA model optimization device 20 based on iterative reinforcement learning can be implemented in the form of a computer program. Figure 3 Runs on the computer device shown.

[0124] See also Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 can be a terminal or a server. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, personal digital assistant, wearable device, or other electronic device with communication capabilities. The server can be a standalone server or a server cluster consisting of multiple servers.

[0125] The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0126] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a VLA model optimization method based on iterative reinforcement learning.

[0127] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0128] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a VLA model optimization method based on iterative reinforcement learning.

[0129] The network interface 505 is used to communicate with other devices over the network. Those skilled in the art will appreciate that the above structure is merely a block diagram of a portion of the structure related to the present invention and does not limit the computer device 500 to which the present invention is applied. A specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0130] The processor 502 is configured to execute a computer program 5032 stored in the memory to implement the following steps:

[0131] Performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder;

[0132] Freezing all parameters of the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model;

[0133] The first optimized VLA model is sent to the server, which unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

[0134] In some preferred embodiments, performing online reinforcement learning on the action decoder to update parameters of the action decoder to obtain a first optimized VLA model includes:

[0135] Collecting environmental images and instruction information, and generating a fusion feature vector based on the environmental images and instruction information;

[0136] Inputting the fused feature vector into the pre-trained VLA model and receiving the robot trajectory data output by the action decoder;

[0137] Determining whether the robot trajectory data is successful trajectory data;

[0138] If the robot trajectory data is successful trajectory data, a preset reinforcement learning algorithm is used based on the robot trajectory data to update the parameters of the action decoder.

[0139] In some preferred embodiments, generating a fused feature vector based on the environment image and the instruction information includes:

[0140] Encoding the environment image into an image feature vector by a preset image encoder;

[0141] Encoding the instruction information into an instruction feature vector by a preset instruction encoder;

[0142] A fusion feature vector is generated according to the image feature vector and the instruction feature vector.

[0143] In some preferred embodiments, determining whether the robot trajectory data is successful trajectory data includes:

[0144] Determining whether the robot trajectory data meets the trajectory standard corresponding to the instruction information;

[0145] If the trajectory data meets the trajectory standard corresponding to the instruction information, the robot trajectory data is determined to be successful trajectory data;

[0146] If the trajectory data does not meet the trajectory standard corresponding to the instruction information, it is determined that the robot trajectory data is not successful trajectory data.

[0147] In some preferred embodiments, the method further comprises:

[0148] If the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data;

[0149] The successful sample data is stored in a successful sample set.

[0150] In some preferred embodiments, performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model includes:

[0151] Acquire an expert sample set and the successful sample set, and generate a training data set based on the expert sample set and the successful sample set;

[0152] The first optimized VLA model is trained through supervised learning using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

[0153] In some preferred embodiments, performing supervised learning training on the first optimized VLA model using the training data set, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model include:

[0154] Collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining an output result of the first optimized VLA model;

[0155] Determining a loss amount based on the output result and a correct result corresponding to the training sample;

[0156] Parameters of the first optimized VLA model are updated according to the loss amount.

[0157] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0158] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0159] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the processor performs the following steps:

[0160] Performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder;

[0161] Freezing all parameters of the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model;

[0162] The first optimized VLA model is sent to the server, which unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

[0163] In some preferred embodiments, performing online reinforcement learning on the action decoder to update parameters of the action decoder to obtain a first optimized VLA model includes:

[0164] Collecting environmental images and instruction information, and generating a fusion feature vector based on the environmental images and instruction information;

[0165] Inputting the fused feature vector into the pre-trained VLA model and receiving the robot trajectory data output by the action decoder;

[0166] Determining whether the robot trajectory data is successful trajectory data;

[0167] If the robot trajectory data is successful trajectory data, a preset reinforcement learning algorithm is used based on the robot trajectory data to update the parameters of the action decoder.

[0168] In some preferred embodiments, generating a fused feature vector based on the environment image and the instruction information includes:

[0169] Encoding the environment image into an image feature vector by a preset image encoder;

[0170] Encoding the instruction information into an instruction feature vector by a preset instruction encoder;

[0171] A fusion feature vector is generated according to the image feature vector and the instruction feature vector.

[0172] In some preferred embodiments, determining whether the robot trajectory data is successful trajectory data includes:

[0173] Determining whether the robot trajectory data meets the trajectory standard corresponding to the instruction information;

[0174] If the trajectory data meets the trajectory standard corresponding to the instruction information, the robot trajectory data is determined to be successful trajectory data;

[0175] If the trajectory data does not meet the trajectory standard corresponding to the instruction information, it is determined that the robot trajectory data is not successful trajectory data.

[0176] In some preferred embodiments, the method further comprises:

[0177] If the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data;

[0178] The successful sample data is stored in a successful sample set.

[0179] In some preferred embodiments, performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model includes:

[0180] Acquire an expert sample set and the successful sample set, and generate a training data set based on the expert sample set and the successful sample set;

[0181] The first optimized VLA model is trained through supervised learning using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

[0182] In some preferred embodiments, performing supervised learning training on the first optimized VLA model using the training data set, updating the parameters of the first optimized VLA model, and obtaining the second optimized VLA model include:

[0183] Collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining an output result of the first optimized VLA model;

[0184] Determining a loss amount based on the output result and a correct result corresponding to the training sample;

[0185] Parameters of the first optimized VLA model are updated according to the loss amount.

[0186] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, etc. Any physical storage medium capable of storing program code can be non-volatile or volatile.

[0187] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0188] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0189] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0190] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0191] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0192] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, to the extent such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to encompass such changes and modifications.

[0193] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A VLA model optimization method based on iterative reinforcement learning, characterized in that: include: Performing supervised learning pre-training on a preset initial VLA model to obtain a pre-trained VLA model, wherein the pre-trained VLA model includes an action decoder; Freezing all parameters of the pre-trained VLA model except the action decoder, performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model; The first optimized VLA model is sent to the server, which unfreezes all parameters of the first optimized VLA model, performs supervised learning training on the first optimized VLA model, updates the parameters of the first optimized VLA model, and obtains a second optimized VLA model.

2. The VLA model optimization method based on iterative reinforcement learning according to claim 1, characterized in that: The performing online reinforcement learning on the action decoder, updating the parameters of the action decoder, and obtaining a first optimized VLA model includes: Collecting environmental images and instruction information, and generating a fusion feature vector based on the environmental images and instruction information; Inputting the fused feature vector into the pre-trained VLA model and receiving the robot trajectory data output by the action decoder; Determining whether the robot trajectory data is successful trajectory data; If the robot trajectory data is successful trajectory data, a preset reinforcement learning algorithm is used based on the robot trajectory data to update the parameters of the action decoder.

3. The VLA model optimization method based on iterative reinforcement learning according to claim 2, characterized in that: The generating of a fusion feature vector based on the environment image and the instruction information includes: Encoding the environment image into an image feature vector by a preset image encoder; Encoding the instruction information into an instruction feature vector by a preset instruction encoder; A fusion feature vector is generated according to the image feature vector and the instruction feature vector.

4. The VLA model optimization method based on iterative reinforcement learning according to claim 2, characterized in that: The determining whether the robot trajectory data is successful trajectory data includes: Determining whether the robot trajectory data meets the trajectory standard corresponding to the instruction information; If the trajectory data meets the trajectory standard corresponding to the instruction information, the robot trajectory data is determined to be successful trajectory data; If the trajectory data does not meet the trajectory standard corresponding to the instruction information, it is determined that the robot trajectory data is not successful trajectory data.

5. The VLA model optimization method based on iterative reinforcement learning according to claim 2, characterized in that: The method further comprises: If the robot trajectory data is successful trajectory data, generating successful sample data based on the successful trajectory data; The successful sample data is stored in a successful sample set.

6. The VLA model optimization method based on iterative reinforcement learning according to claim 5, characterized in that: The performing supervised learning training on the first optimized VLA model, updating the parameters of the first optimized VLA model, and obtaining a second optimized VLA model includes: Acquire an expert sample set and the successful sample set, and generate a training data set based on the expert sample set and the successful sample set; The first optimized VLA model is trained through supervised learning using the training data set, and the parameters of the first optimized VLA model are updated to obtain a second optimized VLA model.

7. The VLA model optimization method based on iterative reinforcement learning according to claim 6, characterized in that: The performing supervised learning training on the first optimized VLA model using the training data set, updating the parameters of the first optimized VLA model, and obtaining a second optimized VLA model includes: Collecting training samples from the training data set and inputting them into the first optimized VLA model, and obtaining an output result of the first optimized VLA model; Determining a loss amount based on the output result and a correct result corresponding to the training sample; Parameters of the first optimized VLA model are updated according to the loss amount.

8. A VLA model optimization device based on iterative reinforcement learning, characterized in that: The method comprises means for performing the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the method according to any one of claims 1 to 7.

Images