A VLA model pre-training method based on multi-scale decoupling of internet video.

Abstract

This application discloses a pre-training method for a VLA model based on multi-scale decoupling from internet video. The method includes: acquiring internet video data and constructing a sequence of action difference frames; extracting features to obtain multi-scale latent features, and obtaining multi-scale decoupling tokens through quantization; reconstructing the internet video data by constructing an inverse dynamics model to spatiotemporally fuse the internet video data and the multi-scale decoupling tokens; simulating the degradation process by adding noise perturbation; constructing a joint loss function to train the model parameters of the multi-scale action token extraction model; and pre-training the visual language action model based on action semantic tokens. This method lowers the data acquisition threshold by directly constructing action difference frame sequences from internet video, replacing expensive robot sensor data, effectively expanding the scale of pre-training data, and alleviating the problem of insufficient data.

Description

Technical Field

[0001] This application relates to the field of computer science, specifically to a VLA model pre-training method based on multi-scale decoupling of Internet video. Background Technology

[0002] The Vision-Language-Action Model (VLA) aims to control robots to perform corresponding actions through a large language model, and has been widely used in robot control, human-computer interaction and other fields.

[0003] However, traditional approaches face numerous challenges in training and application, particularly the issues of insufficient data and the mixing of multimodal information. Insufficient data is a major problem for current visual language action models in robot control, especially for training embodied agents such as robotic arms. High-quality motion data typically requires a large number of sensors and control devices, which are not only expensive but also struggle to cover diverse motion scenarios, resulting in limited training data.

[0004] To address this issue, internet video has become an important alternative data source, providing rich motion and scene information. However, how to efficiently utilize internet video for VLA model pre-training remains a pressing technical challenge.

[0005] Traditional pre-training methods typically rely on single-scale feature extraction, lacking multi-scale abstraction of video information. This single-scale modeling approach can easily lead to overfitting of the model to low-level pixel-level details. The model may directly copy pixel patterns through shortcuts, thus avoiding the extraction of action semantics, resulting in poor generalization performance of the trained model. Summary of the Invention

[0006] To address the aforementioned issues, this application proposes a VLA model pre-training method based on multi-scale decoupling of internet videos, comprising: Acquire internet video data, and construct an action difference frame sequence based on the feature differences between the current frame image in the internet video data and its corresponding subsequent frame images; Based on a multi-scale variational autoencoder, feature extraction is performed on the action difference frame sequence to obtain multi-scale latent features. Then, through quantization, the multi-scale latent features are discretized to obtain a multi-scale decoupling token. By constructing a reverse dynamics model, the Internet video data and the multi-scale decoupling token are spatiotemporally fused to obtain corresponding spatiotemporal features, and the Internet video data is reconstructed based on the spatiotemporal features to obtain reconstructed video data. Based on the introduced noise degradation mechanism, noise perturbation is added to the action semantic tokens used to describe motion patterns in the multi-scale latent features to simulate the degradation process and obtain simulated video first frame data. Based on the reconstructed video data, the simulated video first frame data, the multi-scale latent features, and the multi-scale decoupling token, a joint loss function is constructed to train the model parameters of the multi-scale action token extraction model; the multi-scale action token extraction model includes the multi-scale variational autoencoder and the inverse dynamics model; The visual language action model is pre-trained based on the action semantic tokens used to describe motion patterns output by the multi-scale action token extraction model.

[0007] In one example, based on the feature differences between the current frame image in the internet video data and its corresponding subsequent frame images, an action difference frame sequence is constructed, specifically including: Based on a preset sliding window, the frame image sequence in the Internet video data is slid. Based on each slide, the corresponding feature differences are determined according to the first and last frame images at both ends of the sliding window, and an action difference frame sequence is constructed.

[0008] In one example, based on a multi-scale variational autoencoder, feature extraction is performed on the action difference frame sequence to obtain multi-scale latent features. These multi-scale latent features are then discretized through quantization to obtain a multi-scale decoupling token, specifically including: The multi-scale variational autoencoder is defined as: the first encoder layer with the lowest resolution, the second encoder layer with the medium resolution, and the third encoder layer with the highest resolution. The first encoder layer extracts global static features, and the global static features are discretized through quantization to obtain an identity token for providing global context. Through the second encoder layer, dynamic features that change over time are extracted, and through quantization, the dynamic features are discretized to obtain action semantic tokens for describing motion patterns. The third encoder layer extracts detailed features corresponding to texture, lighting, and local appearance, and discretizes these detailed features through quantization to obtain pixel detail tokens.

[0009] In one example, by constructing a reverse dynamics model, the internet video data and the multi-scale decoupling token are spatiotemporally fused to obtain the corresponding spatiotemporal features, specifically including: Based on the constructed inverse dynamics model, the real video first frame data of the Internet video data and the multi-scale decoupling token are fused through a spatial fusion mechanism to obtain a spatial fusion feature map; Through a time fusion mechanism, the corresponding temporal correlation features are obtained based on the multi-scale decoupling tokens; The spatial fusion feature map and the temporal correlation feature are fused to obtain the corresponding spatiotemporal features.

[0010] In one example, based on the introduced noise degradation mechanism, the degradation process is simulated by adding noise perturbations to the action semantic tokens used to describe motion patterns in the multi-scale latent features, resulting in simulated video first frame count data, specifically including: Based on the noise degradation mechanism, the spatiotemporal features obtained after spatiotemporal fusion are used as the initial state; Starting from the initial state, Gaussian random noise is progressively superimposed on the dynamic features extracted by the corresponding second encoder layer; wherein, the perturbation form of each step is: the action semantic state of the current step is obtained by the action semantic state of the previous step and the Gaussian random noise of the current step; Based on the state sequence composed of the obtained states, a Markov chain is modeled, and based on the Markov chain, when the superimposed Gaussian random noise is greater than a preset threshold, the first frame data of the simulated video is obtained.

[0011] In one example, a joint loss function is constructed based on the reconstructed video data, the simulated video first frame data, the multi-scale latent features, and the multi-scale decoupling token, specifically including: Based on reconstruction loss, codebook loss, and degradation mechanism loss, the total loss function is obtained by weighted combination. The reconstruction loss is obtained based on the degree of difference between the reconstructed video data and the internet video data. The codebook loss is obtained based on the degree of difference between the multi-scale latent features and the most recent multi-scale decoupling token; The degradation mechanism loss is obtained based on the degree of difference between the simulated video first frame data and the real video first frame data corresponding to the Internet video data.

[0012] In one example, the codebook loss is: ;in, For codebook loss, This indicates that the gradient operation is stopped. Let i be the i-th multi-scale latent feature output by the multi-scale variational autoencoder. To and Recent multi-scale decoupling tokens, For weights.

[0013] In one example, the visual language action model is pre-trained based on the action semantic tokens used to describe motion patterns output by the multi-scale action token extraction model, specifically including: Determine the action semantic tokens obtained through the second encoder layer to describe the motion pattern; Align the action semantic token with the corresponding original frame and text description in the Internet video data to pre-train the visual language action model.

[0014] In one example, the method further includes: The acquired robot's actual operational data is smaller than the preset number. The visual language action model is adjusted based on the actual operating data.

[0015] On the other hand, this application also proposes a VLA model pre-training device based on multi-scale decoupling of Internet video, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the VLA model pre-training method based on multi-scale decoupling of Internet video as described in any of the above examples.

[0016] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions configured to implement the VLA model pre-training method based on multi-scale decoupling of Internet video as described in any of the above examples.

[0017] The VLA model pre-training method based on multi-scale decoupling of Internet video proposed in this application can bring the following beneficial effects: 1. Lowering the data acquisition threshold: Directly constructing motion difference frame sequences from internet videos to replace expensive robot sensor data, effectively expanding the scale of pre-training data and alleviating the problem of insufficient data.

[0018] 2. Avoid pixel-level shortcut fitting: Multi-scale variational autoencoders extract latent features of different granularities and discretize them into decoupling tokens, forcing the model to learn hierarchical motion semantics, reducing mechanical copying of low-level pixel patterns, and improving the model's generalization ability.

[0019] 3. Suppressing irrelevant information interference: The inverse dynamics model, through spatiotemporal fusion and video reconstruction constraints, prompts the decoupling token to focus on spatiotemporal changes related to the causality of actions, weakening the confusing factors such as scene appearance and background, and extracting a more purified action representation.

[0020] 4. Enhance robustness: Introduce a noise degradation mechanism on the action semantic token to simulate visual degradation in the real environment, force the model to learn the essential motion pattern, reduce sensitivity to specific video details, and improve adaptability to different scenes and noise conditions.

[0021] 5. Improve downstream task transfer efficiency: Joint loss optimization outputs high-quality, decoupled action semantic tokens for VLA model pre-training, which can inject strong action priors into robot control, enabling VLA to quickly adapt to small sample robot data and improve task execution capabilities in open environments. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the VLA model pre-training method based on multi-scale decoupling of Internet video in the embodiments of this application; Figure 2 This is a schematic diagram of a noise degradation mechanism in one scenario of an embodiment of this application; Figure 3 This is a schematic diagram of a VLA model pre-training device based on multi-scale decoupling of Internet video in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, this application provides a VLA model pre-training method based on multi-scale decoupling of Internet video, including: S101: Acquire Internet video data, and construct an action difference frame sequence based on the feature differences between the current frame image in the Internet video data and its corresponding subsequent frame images.

[0026] Internet video data mainly includes robot-related video data, which can be targeted for specific application scenarios based on the visual language action model (VLA) that needs to be trained. For example, it can be targeted for video data corresponding to embodied robots, wheeled mobile robots, and robotic arms.

[0027] When constructing the motion difference frame sequence, a preset sliding window is used to slide across the frame image sequence in the internet video data. Based on each slide, the corresponding feature differences are determined according to the first and last frame images at both ends of the sliding window, and the motion difference frame sequence is constructed. Specifically, in determining the feature differences, a feature extraction network is first used to convert pixel representations into latent feature representations. Then, the differences between the latent features are used to obtain the corresponding feature differences. Furthermore, if the width of the sliding window is set to two frames, the feature differences represent the differences between two adjacent frames.

[0028] The action difference frame sequence obtained in this way is no longer a difference image at the original pixel level, but a difference vector in an abstract, higher-order latent space.

[0029] S102: Based on a multi-scale variational autoencoder, feature extraction is performed on the action difference frame sequence to obtain multi-scale latent features. Then, through quantization, the multi-scale latent features are discretized to obtain a multi-scale decoupling token.

[0030] Specifically, the multi-scale variational autoencoder comprises: a first encoder layer with the lowest resolution, a second encoder layer with medium resolution, and a third encoder layer with the highest resolution. Each layer aims to extract features at a specific scale and is connected to a discrete codebook for quantization representation. Through this structure, the visual information of the input action difference frame sequence is extracted and discretized according to scale.

[0031] The first encoder layer extracts global static features, which are then discretized through quantization to obtain an identity token that provides global context. This identity token, obtained from the lowest-resolution first encoder layer, is primarily responsible for extracting global, static features from the video. Through vector quantization (also known as discretization), these continuous features are mapped to discrete identity tokens, providing a stable global context.

[0032] The second encoder layer extracts time-varying dynamic features, which are then discretized through quantization to obtain action semantic tokens describing motion patterns. These action tokens, obtained through a mid-resolution second encoder layer, are specifically designed to capture time-varying dynamic features in the video. After quantization, a series of discrete action semantic tokens are generated, designed to encode motion trajectories, behavioral patterns, and the semantics of actions.

[0033] The third encoder layer extracts detailed features corresponding to texture, lighting, and local appearance. These features are then discretized through quantization to obtain pixel detail tokens. These pixel detail tokens, obtained through the highest resolution third encoder layer, are used to capture fine visual details such as high-frequency texture, lighting, and local appearance.

[0034] In the quantization process described above, continuous feature representations are converted into discrete symbolic representations. Here, the features output by the i-th layer encoder can be denoted as... Features at all scales share the same codebook vector set. , where k is the codebook size. The vector quantization operator Q is used to quantize... Mapped to discrete codebook vector ,in .

[0035] S103: By constructing a reverse dynamics model, the Internet video data and the multi-scale decoupling token are spatiotemporally fused to obtain corresponding spatiotemporal features, and the Internet video data is reconstructed based on the spatiotemporal features to obtain reconstructed video data.

[0036] The core function of the inverse dynamics model is to predict and reconstruct the complete video sequence from the decoupled initial frame features and multi-scale tokens.

[0037] Specifically, based on the constructed inverse dynamics model, a spatial fusion mechanism is used to fuse the real first frame data of internet video data and multi-scale decoupling tokens to obtain a spatial fusion feature map. This spatial fusion mechanism integrates the static identity information of the real first frame data with the multi-scale decoupling tokens. For example, the feature map obtained after encoding the real first frame data can be used as a spatial base. The discrete multi-scale decoupling tokens are then retrieved from the codebook, and their corresponding continuous embedding vectors are injected into this feature map according to their spatial location for fusion.

[0038] Through a temporal fusion mechanism, corresponding temporal correlation features are obtained based on multi-scale decoupled tokens. For example, multi-scale decoupled tokens are fed into a temporal modeling module (such as a Transformer or a temporal convolutional network) in chronological order, allowing the model to learn the dependencies and evolutionary patterns between tokens at adjacent time points, ensuring the continuity of actions, thereby obtaining the corresponding temporal correlation features.

[0039] The spatial fusion feature map and temporal correlation features are fused to obtain the corresponding spatiotemporal features. The comprehensive spatiotemporal feature sequence obtained after spatial and temporal fusion is fed into the decoder network, and the pixel images of each frame are reconstructed layer by layer to complete the video reconstruction.

[0040] S104: Based on the introduced noise degradation mechanism, noise perturbation is added to the action semantic tokens used to describe motion patterns in the multi-scale latent features to simulate the degradation process and obtain simulated video first frame data.

[0041] By progressively adding random noise to the action semantic feature layer, the degradation and loss of action information is simulated, and this process is modeled as a Markov chain.

[0042] Specifically, based on the noise degradation mechanism, the spatiotemporal features obtained after spatiotemporal fusion are used as the initial state. The latent representation (also known as the hidden representation) obtained after spatiotemporal fusion is used as the initial state. It contains a complete set of multi-scale decoupling tokens (including identity tokens, action semantic tokens, and detail tokens).

[0043] Starting from the initial state, Gaussian random noise is progressively superimposed on the dynamic features extracted from the corresponding second encoder layer. The perturbation at each step is as follows: the action semantic state of the current step is obtained by combining the action semantic state of the previous step with the Gaussian random noise of the current step. This random perturbation is then introduced. Features applied before action semantic token extraction interfere with the extracted action semantic token, leading to the next state. The perturbation at each step can be formalized as: ,in This is the noise item that was added.

[0044] Based on the state sequence formed by the obtained states, a Markov chain is modeled, and based on the Markov chain, the first frame of the simulated video is obtained when the superimposed Gaussian random noise exceeds a preset threshold. After t iterations of perturbation, a state sequence is formed. , ,... , This constructs an evolutionary path from the original state to the degenerate state, the effect of which is as follows: Figure 2 As shown.

[0045] This sequence of states (also known as the evolution path of degenerate states, or simply the degenerate path) possesses Markov properties and satisfies the following conditions: Therefore, it can be This can be viewed as a Markov chain driven by noise perturbation. As t increases, the effective information contained in the action semantic tokens is gradually submerged by noise, and its representational power monotonically decreases. When the perturbation accumulates to a certain level, almost all action-related information is lost, and the model's state will be mainly supported by identity tokens and residual pixel detail tokens. At this point, the video sequence degenerates into an initial frame with basically no action changes, which is referred to here as the simulated video first frame data. When the noise intensity exceeds a threshold, the action semantic layer is completely destroyed, and the system converges to the first frame state.

[0046] S105: Based on the reconstructed video data, the simulated video first frame data, the multi-scale latent features, and the multi-scale decoupling token, a joint loss function is constructed to train the model parameters of the multi-scale action token extraction model; the multi-scale action token extraction model includes the multi-scale variational autoencoder and the inverse dynamics model.

[0047] Based on reconstruction loss, codebook loss, and degradation mechanism loss, a total loss function is obtained through a weighted combination. The network model parameters are optimized by minimizing this comprehensive loss function. This total loss function is a weighted combination of the three core loss terms. It can be defined as: .

[0048] Among them, reconstruction losses ( The loss term is derived from the degree of difference between the reconstructed video data and the internet video data. It aims to measure the fidelity between the inverse dynamics predicted frame (i.e., the reconstructed video data) and the original frame (i.e., the internet video data).

[0049] Codebook loss ( The loss term is derived based on the degree of difference between the multi-scale latent features and the most recent multi-scale decoupling tokens. This loss term is the VQ-VAE loss, used to optimize the vector quantization (VQ) module. It ensures that the multi-scale latent features output by the encoder can be efficiently quantized into discrete codebook vectors (i.e., multi-scale decoupling tokens), and promotes full utilization of the codebook vectors, as shown in the formula: ;in, For codebook loss, This indicates that the gradient operation is stopped. Let i be the i-th multi-scale latent feature output by the multi-scale variational autoencoder. To and Recent multi-scale decoupling tokens, The weight of the second term. The first term in this loss ( The second term () pulls the features output by the multi-scale variational autoencoder to the nearest codebook vector, while the second term () Then, the codebook vector is updated through backpropagation, making it more similar to the features output by the multi-scale variational autoencoder.

[0050] Degradation mechanism loss ( The degree of difference between simulated video first-frame data and the corresponding real video first-frame data from the internet is obtained. Its supervised inverse dynamics model learns how to reverse the degradation process from noisy action semantic tokens and reconstruct the original frame sequence.

[0051] Weights corresponding to each loss , , Then you can set it based on your actual needs.

[0052] S106: Based on the action semantic tokens used to describe motion patterns output by the multi-scale action token extraction model, the visual language action model is pre-trained.

[0053] Specifically, action semantic tokens obtained through the second encoder layer are determined to describe motion patterns. These action semantic tokens are then aligned with their corresponding original video frames and text descriptions in the internet video data to pre-train the visual language action model. The text descriptions can include action instructions, task objectives, or video content descriptions. The original video frames, text descriptions, and action semantic tokens are aligned along the timeline, enabling the model to learn the action encoding corresponding to each video segment.

[0054] Furthermore, pre-trained visual-language action models have learned general, actuator-independent action semantics from internet videos. However, key fine-tuning processes are still needed to effectively transfer this general capability to real-world embodied intelligence tasks such as robotic arm control.

[0055] Based on this, a smaller-scale set of real-world operational data corresponding to the robot is obtained; the visual language motion model is then adjusted based on this real-world operational data. Using a smaller but higher-quality set of real-world robotic arm data, the pre-trained model is fine-tuned to ensure it can adapt to the actual actuator and possess reliable motion control capabilities.

[0056] 1. Lowering the data acquisition threshold: Directly constructing motion difference frame sequences from internet videos to replace expensive robot sensor data, effectively expanding the scale of pre-training data and alleviating the problem of insufficient data.

[0057] 2. Avoid pixel-level shortcut fitting: Multi-scale variational autoencoders extract latent features of different granularities and discretize them into decoupling tokens, forcing the model to learn hierarchical motion semantics, reducing mechanical copying of low-level pixel patterns, and improving the model's generalization ability.

[0058] 3. Suppressing irrelevant information interference: The inverse dynamics model, through spatiotemporal fusion and video reconstruction constraints, prompts the decoupling token to focus on spatiotemporal changes related to the causality of actions, weakening the confusing factors such as scene appearance and background, and extracting a more purified action representation.

[0059] 4. Enhance robustness: Introduce a noise degradation mechanism on the action semantic token to simulate visual degradation in the real environment, force the model to learn the essential motion pattern, reduce sensitivity to specific video details, and improve adaptability to different scenes and noise conditions.

[0060] 5. Improve downstream task transfer efficiency: Joint loss optimization outputs high-quality, decoupled action semantic tokens for VLA model pre-training, which can inject strong action priors into robot control, enabling VLA to quickly adapt to small sample robot data and improve task execution capabilities in open environments.

[0061] like Figure 3 As shown in the illustration, this application also provides a VLA model pre-training device based on multi-scale decoupling of Internet video, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the VLA model pre-training method based on multi-scale decoupling of Internet video as described in any of the above embodiments.

[0062] This application also provides a non-volatile computer storage medium storing computer-executable instructions configured to implement the VLA model pre-training method based on multi-scale decoupling of Internet video as described in any of the above embodiments.

[0063] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0064] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0065] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A VLA model pre-training method based on multi-scale decoupling of Internet video, characterized in that, include: Acquire internet video data, and construct an action difference frame sequence based on the feature differences between the current frame image in the internet video data and its corresponding subsequent frame images; Based on a multi-scale variational autoencoder, feature extraction is performed on the action difference frame sequence to obtain multi-scale latent features. These latent features are then discretized through quantization to obtain a multi-scale decoupling token. Specifically, the process includes: determining that the multi-scale variational autoencoder comprises a first encoder layer with the lowest resolution, a second encoder layer with a medium resolution, and a third encoder layer with the highest resolution; extracting global static features through the first encoder layer and discretizing these features through quantization to obtain an identity token providing global context; extracting time-varying dynamic features through the second encoder layer and discretizing these dynamic features through quantization to obtain an action semantic token describing motion patterns; and extracting detail features corresponding to texture, lighting, and local appearance through the third encoder layer and discretizing these detail features through quantization to obtain a pixel detail token. By constructing a reverse dynamics model, the Internet video data and the multi-scale decoupling token are spatiotemporally fused to obtain corresponding spatiotemporal features, and the Internet video data is reconstructed based on the spatiotemporal features to obtain reconstructed video data. Based on the introduced noise degradation mechanism, noise perturbation is added to the action semantic tokens used to describe motion patterns in the multi-scale latent features to simulate the degradation process and obtain simulated video first frame data. Specifically, this includes: using the spatiotemporal features obtained after spatiotemporal fusion as the initial state based on the noise degradation mechanism; starting from the initial state, Gaussian random noise is progressively superimposed on the dynamic features extracted by the corresponding second encoder layer; wherein the perturbation form of each step is: the action semantic state of the current step is obtained by the action semantic state of the previous step and the Gaussian random noise of the current step; a Markov chain is modeled based on the state sequence composed of the obtained states, and simulated video first frame data is obtained based on the Markov chain when the superimposed Gaussian random noise is greater than a preset threshold. Based on the reconstructed video data, the simulated video first frame data, the multi-scale latent features, and the multi-scale decoupling token, a joint loss function is constructed to train the model parameters of the multi-scale action token extraction model; the multi-scale action token extraction model includes the multi-scale variational autoencoder and the inverse dynamics model; The visual language action model is pre-trained based on the action semantic tokens used to describe motion patterns output by the multi-scale action token extraction model.

2. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 1, characterized in that, Based on the feature differences between the current frame image in the internet video data and its corresponding subsequent frame images, an action difference frame sequence is constructed, specifically including: Based on a preset sliding window, the frame image sequence in the Internet video data is slid. Based on each slide, the corresponding feature differences are determined according to the first and last frame images at both ends of the sliding window, and an action difference frame sequence is constructed.

3. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 1, characterized in that, By constructing a reverse dynamics model, the internet video data and the multi-scale decoupling token are spatiotemporally fused to obtain the corresponding spatiotemporal features, specifically including: Based on the constructed inverse dynamics model, the real video first frame data of the Internet video data and the multi-scale decoupling token are fused through a spatial fusion mechanism to obtain a spatial fusion feature map; Through a time fusion mechanism, the corresponding temporal correlation features are obtained based on the multi-scale decoupling tokens; The spatial fusion feature map and the temporal correlation feature are fused to obtain the corresponding spatiotemporal features.

4. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 1, characterized in that, Based on the reconstructed video data, the simulated video first frame data, the multi-scale latent features, and the multi-scale decoupling token, a joint loss function is constructed, specifically including: Based on reconstruction loss, codebook loss, and degradation mechanism loss, the total loss function is obtained by weighted combination. The reconstruction loss is obtained based on the degree of difference between the reconstructed video data and the internet video data. The codebook loss is obtained based on the degree of difference between the multi-scale latent features and the most recent multi-scale decoupling token; The degradation mechanism loss is obtained based on the degree of difference between the simulated video first frame data and the real video first frame data corresponding to the Internet video data.

5. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 4, characterized in that, The codebook loss is: ;in, For codebook loss, This indicates that the gradient operation is stopped. Let i be the i-th multi-scale latent feature output by the multi-scale variational autoencoder. To and Recent multi-scale decoupling tokens, For weights.

6. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 1, characterized in that, Based on the action semantic tokens used to describe motion patterns output by the multi-scale action token extraction model, the visual language action model is pre-trained, specifically including: Determine the action semantic tokens obtained through the second encoder layer to describe the motion pattern; Align the action semantic token with the corresponding original frame and text description in the Internet video data to pre-train the visual language action model.

7. The VLA model pre-training method based on multi-scale decoupling of Internet video according to claim 1, characterized in that, The method further includes: The acquired robot's actual operational data is smaller than the preset number. The visual language action model is adjusted based on the actual operating data.

Images