A video pre-training method and action prediction method based on graph generation

Abstract

The present disclosure provides a video pre-training method and prediction method based on graph generation. The pre-training method comprises: constructing a generation graph containing object vertices and visual action vertices based on an observed image; estimating object attributes using a graph and its history graph; inputting the graph with object attributes into a graph neural network to generate vertex features; predicting a future graph sequence based on image encoding, language instruction encoding and the vertex features using a graph generator; training the network involved in the above steps using a video dataset to obtain a graph generation model; using the graph generation model, predicting the future graph sequence of the agent's perspective observation image using an agent demonstration dataset with action labels; and adding the future graph sequence to the agent demonstration dataset with action labels to train a graph-guided strategy. Using the present application, a transferable graph with rich action information can be constructed within a video frame, and a future graph can be generated to provide accurate and expressive guidance for graph-guided strategy training.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent technology, specifically to a video pre-training method and an action prediction method based on graph generation. Background Technology

[0002] Imitation learning from demonstrations has become an effective paradigm for teaching robots to perform various behaviors. However, collecting demonstrations typically requires significant human input, which has become a bottleneck for robot learning. Faced with this challenge, video has emerged as an alternative, providing a rich data source containing behavioral and physical knowledge. Previous research has explored various methods for utilizing video data, including visual pre-training and learning reward functions. However, these methods overlook the dynamics of transformation or face significant gaps between simulation and reality. Recent research attempts to address these issues by pre-training models on video data to predict subsequent images or points from current images, and then using these predictions to guide policy learning on data with action labels.

[0003] While current methods have shown potential, they model representations directly from pixel space, neglecting the modeling of important structures, such as the underlying internal structure of objects, spatial relationships between objects, and relationships between objects and actuators. Summary of the Invention

[0004] In view of this, the present invention provides a video pre-training method based on graph generation, which can construct transferable graphs rich in action information within video frames and generate future graphs to provide accurate and expressive guidance.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0006] A graph generation-based video pre-training method includes:

[0007] Step 1: Construction of transferable graphs with enhanced action information: For the observed image o t Obtain object vertices by performing object segmentation Perform key interaction point estimation to obtain visual action vertices. With object vertex and visual action vertex Forming a vertex set and generating a graph

[0008] Step 2: Graph Generation and Modeling: Using the graph... Its history graph estimates object attributes, and the embedding graph The vertices of the object in the graph; the graph to which object attributes will be added. Input the first graph into a neural network to generate vertex features.

[0009] Step 3: Graph Generation: Using a graph generator, generate the graph from the observed image o. t Image encoding Language instruction encoding and the vertex features Integrate and generate graph features In turn, the future will be generated. Step-by-step graph sequence

[0010] Step 4: Utilize the video dataset Observe the image and language instructions, execute steps 1-3, train the network used in steps 2 and 3, and obtain the graph generation model.

[0011] Preferably, in step 1, the object segmentation of the observed image to obtain object vertices... for:

[0012] For time step t in the task process, use the segmentation model to predict the currently observed image o. t The object mask is used to uniformly sample within the predicted object mask to obtain the object vertices.

[0013] Preferably, in step 1, the key interaction point estimation is performed to obtain the visual action vertex. For: From the observed image o t Extract the fingertip position as the key interaction point to obtain the visual action vertex.

[0014] Preferably, in step 2, the use of the diagram The object attributes estimated from its historical graph are as follows:

[0015] For the object Ω in the observed image i From the figure Extracting object images

[0016] From the H-step historical map Searching for objects Ω i Centered object diagram

[0017] Based on object graph and Using online tracking algorithms to obtain object Ω from historical observations i vertex set Vertex attributes include positional information and semantic category;

[0018] Object Ω i Historical position information of the k-th vertex and semantic category attributes Mapped to first vertex embedding

[0019] The object image and the first vertex embedding of each vertex Input the second graph into the neural network to obtain features. Average pooling is performed between vertices to form object graph features.

[0020] Based on object graph features Generate object attribute features

[0021] Preferably, the object graph feature-based Generate object attribute features for:

[0022] Object graph features The hidden embedding h is input into the Long Short-Term Memory (LSTM) module and outputs the hidden embedding h. t Mapped to object attribute features

[0023] Preferably, the graph generation modeling in step 2 further includes multi-level graph modeling, specifically including:

[0024] In the figure Introducing a global vertex layer in the graph: medium object Ω i The vertices of the objects are leaf vertices. K-means clustering is used to divide them into non-overlapping clusters based on their spatial positions. A root global vertex is added to each cluster, and its spatial position is the average of the spatial positions of all leaf vertices in that cluster. All objects Ω i The root global vertex constitutes the global vertex layer;

[0025] Construct edges to connect the hierarchical structure: pairs of directed edges between all root global vertices and pairs of directed edges from each leaf vertex to the corresponding root global vertex;

[0026] The root global vertex and leaf vertices belonging to the same object are assigned the same semantic category and object attributes, forming a multi-level graph;

[0027] Step 2 will add the graph containing the object's attributes. Input the first graph into a neural network to generate vertex features. for:

[0028] The location information, semantic category attributes, and estimated object attributes from the past H time steps are integrated and mapped into a second vertex embedding. This embedding is then aggregated along the edges of the multi-level graph using a first graph neural network to obtain vertex features.

[0029] Preferably, in step 3, an image generator is used to process the observed image o. t Image encoding Language instruction encoding and the vertex features Integrate and generate graph features In turn, the future will be generated. Step-by-step graph sequence for:

[0030] BERT is used to encode the language text to obtain the language instruction encoding.

[0031] The observed image is encoded using a ResNet network to obtain the image encoding.

[0032] Image encoding using transformer Language instruction encoding and the vertex features Integrate to obtain graph features Graphic features The input is fed into the prediction head of a two-layer multilayer perceptron (MLP) to generate future graph sequences.

[0033] Preferably, the mapping employs an MLP.

[0034] The present invention also provides an action prediction method based on graph generation, comprising:

[0035] Graph-guided strategy training steps: Use a demonstration dataset of agents with action labels. Using the graph generation model obtained by any one of the video pre-training methods in claims 1-8, predict the observed image from the agent's perspective. t Future graph sequence With future diagram sequence Observe image o t and agent state ρ t As input, action labels serve as the ideal output, and the training graph guides the strategy.

[0036] Prediction step: The agent acquires the currently observed image. t and agent state ρ t The graph generation model is used to generate the future. Step-by-step graph sequence With the generated graph sequence Current observation image o t and agent state ρ t As input, a pre-trained graph-guided strategy is used to predict a series of actions of the agent.

[0037] Preferably, in the graph-guided strategy training step, future graph sequences are used. Observe image o t and agent state ρ t As input, action labels are the ideal output; the training graph guidance strategy is as follows:

[0038] A state encoder composed of transformer blocks is used to process the graph sequence at time step t. Observe image o t and agent state ρ t Encode to obtain the corresponding projection mark. Use a transformer to generate the state representation s t ;

[0039] Let the state be represented as s t With the projection marker of the graph sequence Connect to obtain policy conditions

[0040] Using MLP action head based Generate action sequences;

[0041] The loss is calculated using the generated action sequence and action labels. The state encoder and MLP action head in the graph-guided policy are then trained via backpropagation to complete the training of the graph-guided policy.

[0042] Beneficial effects:

[0043] (1) This invention proposes a novel framework, GraphMimic, which connects video pre-training with policy learning through graph-to-graph generative modeling. GraphMimic constructs transferable graphs rich in action information within video frames, avoiding problems such as fuzzy spatial relationship modeling and inaccurate future predictions caused by modeling only pixel space. Simultaneously, it generates future graphs to provide accurate and expressive guidance. With the help of future graphs, policy learning can be achieved on limited action-labeled data.

[0044] (2) In constructing the graph generation model, this invention introduces key interaction points abstracted from agents, enabling it to capture important object-agent interactions and avoid the morphological heterogeneity between different agents hindering direct motion transfer. By adding key interaction points, the graph generation network trained on video images in this invention learns the general principles of object interaction, focusing on the functional aspects of operation. The trained graph generation network is also applicable to agents of various forms, promoting cross-agent transfer.

[0045] (3) This invention proposes an object attribute estimation method to facilitate the prediction of behavior of diverse objects.

[0046] (4) In a preferred embodiment of the present invention, in order to simulate long-range dependencies to capture spatial relationships, the present invention introduces an additional hierarchical level in the construction graph to facilitate direct long-range message propagation. At the same time, this hierarchical architecture can capture local structures and spatial relationships, and can effectively model important internal structures while capturing long-range dependencies.

[0047] (5) Extensive experiments across more than 140 tasks show that the method of this invention outperforms state-of-the-art methods by more than 17% and 23% in simulated and real-world experiments, respectively, and demonstrates a performance improvement of more than 33% in cross-existence transfer experiments. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the generation of future graphs using the video pre-training method in GraphMimic of this invention.

[0049] Figure 2 This is a schematic diagram of the robot action prediction based on the generation of future graphs in the GraphMimic of this invention.

[0050] Figure 3 This is a schematic diagram of the object attribute prediction method of the present invention.

[0051] Figure 4 This is a schematic diagram of the multi-layer diagram construction of the present invention. Detailed Implementation

[0052] This invention provides a graph-based video pre-training method that models video pre-training as a graph-to-graph generative model. Graph representation essentially encodes spatial relationship biases, enabling effective structural modeling. During modeling, not only are graphs built for objects in video frames, but also visual action vertices are incorporated into the constructed graph to capture the relationships between objects and actuators (e.g., a robotic arm). Object attributes are also added to the graph structure. Using this method, transferable graphs rich in action information can be constructed within video frames using video datasets, avoiding problems such as vague spatial relationship modeling and inaccurate future predictions caused by modeling only pixel space.

[0053] Although the aforementioned transferable graph is based on video graphs, it contains object structure information and abstracts hand movements into visual action vertices. Therefore, the constructed graph is also applicable to action-labeled agent demonstration datasets. After training the graph generation model using the video dataset, during graph-guided strategy training, the graph generation model can be used to generate future graph sequences from the agent's current observation image, and then based on the agent's current observation image... t The system uses future graphs, agent states, and known action labels to learn policies on limited action-labeled data.

[0054] The scheme consisting of the above-mentioned pre-training method and prediction method is called GraphMimic.

[0055] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] Consider a dataset of motionless videos And robot demonstration datasets with action labels Among them l i It is the language instruction of the i-th segment. This represents an observation sequence consisting of camera images. These are sequences with action labels. The video dataset is used to train the graph generation model, and the robot demonstration dataset with action labels is used to train the graph guidance strategy.

[0057] The objective of this invention is to utilize readily available, unlabeled video datasets. Pre-training is performed to learn a policy that mimics expert behavior. The overall process of GraphMimic is as follows: Figure 1 and Figure 2 As shown, it includes two training phases. The first phase utilizes the video dataset. The training of the graph generation model involves two stages. First, a graph structure is constructed within video frames. Based on this graph structure, object attributes are predicted, and a multi-level structural graph is built to capture important local structures and spatial relationships. Then, a pre-trained generative modeling network is used to generate future graphs based on the constructed graph, thus enabling the prediction of multiple future graphs from a single graph. The second stage utilizes a robot demonstration dataset with action labels. A graph guidance policy is trained using future graphs generated by a graph generation model. The graph guidance policy network predicts robot actions based on these generated future graphs by learning from demonstrations with action labels using few samples.

[0058] Each step is described in detail below.

[0059] Step 1: Transferable graph with enhanced motion information.

[0060] In this step, the observed image o is based on the task description l and time step t. t Constructing a graph representation in, Represented by object vertices and visual action vertices The vertex set composed of ε t Let Ω represent the set of edges representing pairs of interacting vertices, and let Ω represent the set of objects in the image.

[0061] Among them, the object vertices The architecture is constructed as follows: the object graph dynamically describes the changes in object states, independent of the agent, thus enhancing GraphMimic's cross-representation learning capabilities. In the initial stage of task execution, a visual language model is used to derive and compile task-related object Ω into text information based on the text instruction l and the initial observed image o0. Subsequently, for each time step t in the task process, an efficient segmentation model is used to predict the current observed image o. t The corresponding object masks are then used to obtain the vertex set by applying a uniform sampling strategy within these predicted masks. in It is an object Ω i The number of vertices. Each vertex's attributes must include at least its position and semantic category. For example... Figure 1 The observed image contains two objects: one is cloth, whose vertices have attributes including fixed position and semantic category "cloth"; the other is the target triangular region of the cloth folded over, whose vertices have attributes including fixed position and semantic category "black triangle".

[0062] Visual Action Vertex The invention is constructed as follows: the introduction of action vertices enables it to capture important object-agent interactions. However, morphological heterogeneity between different agents hinders direct motion transfer. To overcome this challenge, the invention abstracts agents into key interaction points, capturing their topological information with objects. For human videos (a primary source of video data), fingertip positions are extracted as key interaction points. Where N a This represents the number of vertices at key interaction points. Previous research indicates that the fingertip is the most frequently touched area during object manipulation. This approach enables GraphMimic to learn general principles of object interaction, focusing on the functional aspects of manipulation and facilitating cross-representational transfer.

[0063] Based on the constructed vertices, edges are connected to generate a graph. Using the objects and the vertices of the visual action, edges are constructed to obtain the current observation graph representation. In one specific implementation, bidirectional edges are established between vertex pairs with a distance less than a threshold d. The transferable graph representation facilitates the effective learning of operational skills from cross-representation videos. Furthermore, it provides meaningful spatial relational biases for graph generation modeling networks, enhancing structural modeling capabilities and thus improving the accuracy of graph generation.

[0064] Step 2: Graph generation and modeling.

[0065] In this step, the product generated in step one is used. Its history graph estimates object attributes, and the embedding graph The vertices of the object in the graph; the graph to which object attributes will be added. Input the first graph into a neural network to generate vertex features.

[0066] Object attribute estimation steps: Objects with different attributes often exhibit different motion patterns. A straightforward way to instill this awareness in the network is through conditional settings based on manually provided parameters (such as stiffness). However, most demonstration data lacks accurate numerical values, hindering model training. Furthermore, manually defining these parameters also impedes the rapid and easy deployment of the network. Instead, this invention proposes a concise and automatic method that infers learned object attributes using a series of consecutive past observations.

[0067] Obtaining object properties targets the fabric and triangular regions within the observed image. For example... Figure 3 As shown, for the object Ω in the observed image i From the figure Extracting object Ω i The corresponding part is called the object diagram. From the historical H-step diagram Searching for objects Ω i Centered object diagram vertex set Only contains object Ω i Vertices and edge sets on Limited to both endpoints corresponding to Ω i The edges. Then, based on the object graph.

[0068] and Using online tracking algorithms to obtain object Ω from historical observations i vertex set Vertex attributes include positional information and semantic category. Subsequently, this is combined with historical positional information. and vertex attributes (representing vertex v) k,t The semantic category is mapped to the k-th vertex. vertex embedding Next, the object diagram and the first vertex embedding of each vertex Inputting an L-layer graph neural network (GNN) performs iterative message passing, aggregating information along the established edges to obtain features. Average pooling is performed between vertices to form object graph features. As shown below:

[0069]

[0070] Equation (1) above indicates that historical location information is processed using MLP. and vertex attributes Mapping to vertex embedding Equation (2) above indicates that GNN is used based on object graphs. and vertex embedding Generate features Equation (3) above represents feature-based Average pooling is performed between vertices to obtain

[0071] Finally, based on object graph features Generate object attribute features In a preferred embodiment, in order to establish long-range time dependencies and improve the accuracy of attribute prediction, features The input is fed into a Long Short-Term Memory (LSTM) module, and the output hidden embeddings are projected onto the attribute features. As shown below:

[0072]

[0073] Among them, h t This represents the hidden embedding at time step t, while c t This represents the cell embedding at time step t. Equation (4) above represents the object graph features. The hidden embedding h generated in the previous iteration t-1 and unit embedding c t-1 The data is input into the LSTM module to generate the hidden embedding at time step t; Equation (5) above indicates that the hidden embedding h is generated using an MLP. t Mapped to object attribute features Object properties and characteristics of all objects It consists of

[0074] Multi-level graph modeling steps: Estimated attribute features It needs to be integrated into the constructed graph In the graph generation modeling network, local structure and spatial relationships within the graph are then captured to predict future graphs. To simulate long-range dependencies and capture spatial relationships, this invention constructs a graph... An additional layer—the global vertex layer—is introduced to facilitate direct long-range message propagation. For each action and object vertex set, such as... Figure 4 As shown in the figure medium object Ω i The object vertices are leaf vertices. The K-means clustering algorithm is used to divide the vertices into non-overlapping clusters based on their spatial location, and a root global vertex is added to each cluster. Its spatial position is the average of the spatial positions of all leaf vertices within the cluster. All objects Ω iThe root global vertices constitute the global vertex layer.

[0075] Then, edges are constructed to connect the hierarchical structure: (I) Edge set (I) An edge set containing directed edges from each leaf vertex to the corresponding root global vertex; (III) A set of directed edges from each root to its leaf vertices; and (III) a set of paired directed edges between all root global vertices. The resulting multi-level diagram The structure is as follows:

[0076]

[0077] In a multi-level graph, the root global vertex and leaf vertices belonging to the same object are assigned the same semantic category and object attributes.

[0078] The constructed multi-level graph is embedded using a GNN network. Position information from the past H time steps is included. Vertex attributes and estimated attribute φ k,t The information is integrated and mapped to vertex embeddings. Then, an L-layer GNN network aggregates the information along the established edges to derive the resulting vertex features. As shown below:

[0079]

[0080] Equation (6) above represents the use of MLP to extract the location information of the past H time steps. Vertex attributes and estimated attribute φ k,t Mapping to vertex embedding Equation (7) above represents a multi-level diagram. and vertex embedding Input GNN to generate vertex features

[0081] Step 3: Graph generation.

[0082] This step utilizes a graph generator to analyze the observed image o. t Image encoding Language instruction encoding and the vertex features Integrate and generate graph features In turn, the future will be generated. Step-by-step graph sequence

[0083] Specifically, in this embodiment, the transformer-based generator integrates graph embedding. Image embedding and language instruction embedding Generated graph features Among them, such as Figure 1 As shown, text conditions are encoded using pre-trained BERT to obtain language instruction embeddings. The observed images are encoded using ResNet to obtain image embeddings. Image Embedding This leads to the acquisition of vertex features in step two. Subsequently, graph features... It is fed into a two-layer MLP prediction head to generate future... Step-by-step graph sequence Since the association of edges directly depends on the positional relationships between vertices, the process of generating a graph sequence can be simplified to generating the spatial positions of vertices. The edge set is derived based on the spatial distance threshold d, thus obtaining the future graph.

[0084] Step 4: Training the graph generation model.

[0085] Steps one through three describe the input and processing procedures of the graph generation model. The networks that need to be trained in the graph generation model include GNN network, graph generator, ResNet network, MPL, and LSTM network.

[0086] This step utilizes video datasets. Observe the images and language commands, execute steps 1-3, and train the network used in steps 2 and 3 to obtain a graph generation model. The graph generation model is labeled with a video dataset. o in t Subsequent video images.

[0087] At this point, the training of the graph generation model is complete.

[0088] Based on the above video pre-training method, this invention also provides a graph generation-based action prediction method, including:

[0089] Step 1: Training the graph-guided strategy.

[0090] In this step, during graph-guided policy training, a demonstration dataset of agents with action labels is used. Using the graph generation model obtained in step 4, the current observed image o from the agent's perspective is generated. t Predicting future graph sequences With graph sequence Current observation image o t and agent state ρ t As input, action labels serve as the ideal output, and the training graph guides the strategy.

[0091] The graph guidance strategy of this invention generates graphs. Current observation o t and robot state ρ t As input, predict a series of actions like Figure 4 As shown, the graph guidance strategy includes a state encoder and an action head.

[0092] This step includes the following sub-steps:

[0093] Step 11: Demonstration dataset of agents with action labels Including the currently observed image from the agent's perspective. t Language commands and action tags; the currently observed image from the agent's perspective. t Using language instructions and following steps one through three, the trained graph generation model is employed to generate future graph sequences.

[0094] Step 12: State Encoder Processing: The state encoder consists of standard transformer blocks, which process all modes at time step t. Encode the state to generate a state representation s. t In this process, a modality-specific encoder projects each modality into a shared embedding space, where a GNN is used to encode the graph and a ResNet is used to process the observed image. t And use MLP to encode robot state ρ t Obtain projection tags across all modalities and views. The learned location embeddings for a specific modality are added together and concatenated with the learned CLS tags. Self-attention is then performed to aggregate information onto the CLS tags, and the resulting CLS tags are used as state representations. t .

[0095] Step 13: Action Head Processing: This step processes the state at each time step independently and parameterizes the action using the MLP action head. To effectively integrate the guiding information from the prediction graph sequence, the state CLS is labeled s. t With diagram label Connect to obtain policy conditions Then, using the MLP action head based on Generate action sequences

[0096] Step 14: Calculate the loss for the generated action sequence and action labels, and backpropagate the MLP, GNN, ResNet and MLP action head in the state encoder of the graph-guided policy.

[0097] Based on an action-labeled agent demonstration dataset Through steps 11-14, the training of the graph-guided strategy is completed.

[0098] As can be seen, the training of this invention consists of two parts: the graph generation model is pre-trained on a video dataset. A cotracker is used to track the graph vertices throughout the video, obtaining their spatial locations and deriving the graph sequence. A loss function is used to calculate the predicted vertex positions. Compared with the true value The mean squared error in pixel space. Then, the graph-guided policy is trained on a few-sample dataset with labeled robot actions. The actions... With the tracking of future map sequences Observe o t and robot state ρ t Combine and construct a training dataset The graph-conditional policy is trained using MSE loss.

[0099] Step 2: Action prediction steps.

[0100] This step specifically includes:

[0101] The agent acquires the currently observed image. t and agent state ρ t ;

[0102] Generate the future using a trained graph generation model Step-by-step graph sequence

[0103] With the generated graph sequence Current observation image o t and agent state ρ t As input, the graph guidance strategy trained using the aforementioned method is used to predict a series of actions of the agent.

[0104] The following experiment will verify the effectiveness.

[0105] 1. Comparison Methods

[0106] This invention, GraphMimic, is compared with four representative methods: (1) BC, representing ordinary behavior cloning, which trains the policy solely based on expert demonstrations with action labels. It uses the same policy architecture as GraphMimic but without graph conditioning, instead using language embeddings to specify the task; (2) R3M, which uses contrastive learning to align video and language representations. Pre-trained Ego4D weights are used, and these weights are fine-tuned on a video dataset to initialize the visual encoder of the BC policy; (3) UniPi, which trains a text-conditional video diffusion model to generate fine-grained video plans, and then trains an inverse dynamics model based on data with action labels; (4) ATM, which trains a trajectory generation model and a corresponding trajectory guidance policy for action prediction.

[0107] 2. Video pre-training

[0108] The first step is to investigate GraphMimic's ability to provide accurate and expressive state representations to improve robot task performance and reduce data requirements.

[0109] Simulation setup: The evaluation here utilizes the LIBERO benchmark, which includes 130 operational tasks under linguistic conditions. This benchmark comprises five different suites. Comparisons are made against a baseline method on each suite. The baseline method is trained for each task using 10 demonstration trajectories with action labels and 50 videos of the robot without action. For each task, each method is trained using three different random seeds.

[0110] Simulation Results: Table 1 shows the average success rate of the LIBERO benchmark. Fifty videos were collected for pre-training, and 'number of L-demos' represents the number of action-labeled demos used for training. The results are the average of three seeds. A summary of the performance comparisons is shown in Table 1, demonstrating that our method outperforms all other methods across all task suites. Our method, trained with only 2 action-labeled trajectories per task, surpasses the state-of-the-art ATM and UniPi methods, which use more than five times the number of action-labeled trajectories. Furthermore, our GraphMimic method achieves superior performance even compared to BC trained on the full dataset. When trained with 10 action-labeled trajectories per task, our method significantly outperforms all baselines, achieving an improvement of over 17%.

[0111] Table 1. Average success rate of LIBERO benchmark tests

[0112]

[0113] Real-world experimental setup: For each task, 10 motion-labeled trajectories and 50 motionless demonstration videos were collected. The policy was trained on all training data using task descriptions as linguistic conditions. Success criteria were assessed manually, with the success rate calculated from the positions and orientations of 10 randomized objects.

[0114] Real-world experimental results: Table 2 shows the success rate of real-world operational experiments. Fifty videos were collected for pre-training, and 'number of L-demos' represents the number of action-labeled demonstrations used for training. The quantitative results in Table 2 demonstrate that GraphMimic consistently and significantly outperforms BC and other video pre-training baselines. Notably, even using only the 20% of action-labeled data required by the baselines, GraphMimic maintains superior performance, consistently outperforming other video pre-training methods. Furthermore, our method exhibits strong fault recovery capabilities. We argue that graph modeling of the topological relationships between objects and actions enables GraphMimic to learn general principles of object interactions, resulting in a significant performance improvement.

[0115] Table 2 Success Rate of Real-World Operational Experiments

[0116]

[0117] 3. Cross-scene migration

[0118] This section explores GraphMimic's ability to provide easily transferable operational knowledge to facilitate learning from cross-representation video data.

[0119] Robot transfer experiment setup: Cross-robot transfer experiments were conducted on the LIBERO benchmark. For each task, we collected 10 motion-labeled demonstration trajectories for UR robotic arms and 50 motionless video demonstration trajectories for Franka robotic arms. The baseline method was validated using UR robotic arms.

[0120] Robotic Transfer Experiment Results: The experimental results of the baseline method and the method of this invention are presented in Table 3, which shows the inter-robot experimental results on the LIBERO benchmark. For each task, 50 videos were collected from the Franka robotic arm, and 10 action-labeled demonstrations were collected from the UR robotic arm. 'L-demos number' represents the number of action-labeled demonstrations used for training. The results are the average of the three seeds. Table 3 shows that the GraphMimic method of this invention still significantly outperforms the comparative methods. The method of this invention exhibits superior performance when trained with only two action-labeled trajectories per task, and achieves an excellent success rate of 70% when using ten trajectories, while UniPi and ATM show significant performance degradation under these conditions. This difference is attributed to the over-inclusion of information specific to a particular representation, hindering cross-representation generalization ability. The GraphMimic method of this invention focuses on object dynamics and key interaction points, enabling it to provide meaningful motion information while enhancing its ability to learn from cross-representation videos.

[0121] Table 3. Results of inter-robot experiments on the LIBERO benchmark.

[0122]

[0123] Human-to-Robot Transfer Experiment Setup: The human-to-robot transfer experiment was conducted in a real-world environment. For each task, we collected 10 motion-tagged trajectories of the Franka robotic arm and 50 motionless videos from humans.

[0124] Human-to-Robot Transfer Experiment Results: Human videos constituted the primary data source for cross-representation video learning. The experimental results are summarized in Table 4, which presents the real-world experimental results for human-to-robot transfer. We collected 50 videos of humans performing the task and 10 robot demonstrations with action labels for each task. 'L-demos number' represents the number of action-labeled demonstrations used for training. Table 4 shows that GraphMimic effectively facilitates the transfer of operational knowledge from human videos to the robotic system, providing expressive guidance for control policy learning. Our method demonstrates better performance compared to other video pre-training methods, confirming its ability to provide effective state representations for cross-representation video learning.

[0125] Table 4. Results of real-world experiments on human-to-robot migration

[0126]

[0127] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A video pre-training method based on graph generation, characterized in that, include: Step 1: Construction of transferable graphs with enhanced action information: [This involves examining the observed image.] Obtain object vertices by performing object segmentation Perform key interaction point estimation to obtain visual action vertices. ; by object vertex and visual action vertex Forming a vertex set and generating a graph ; Step 2: Graph Generation and Modeling: Using the graph... Its history graph estimates object attributes, and the embedding graph The vertices of the object in the graph; the graph to which object attributes will be added. Input the first graph into a neural network to generate vertex features. ; Step 3: Graph Generation: Using a graph generator, generate the observed image. Image encoding Language instruction encoding and the vertex features Integrate and generate graph features And thus generate the future Step-by-step graph sequence ; Step 4: Utilize the video dataset Observe the image and language instructions, execute steps 1-3, train the network used in steps 2 and 3, and obtain the graph generation model; In step 2, the use of the diagram The object attributes estimated from its historical graph are as follows: For objects in the observed image From the figure Extracting object images ; from H Step by step historical map Search by object Centered object diagram ; Based on object graph and Using online tracking algorithms to obtain objects from historical observations vertex set Vertex attributes include position information and semantic category; Object The Middle k Historical position information of each vertex and semantic category attributes Mapped to first vertex embedding ; The object image and the first vertex embedding of each vertex Input the second graph into the neural network to obtain features. Average pooling is performed between vertices to form object graph features. ; Based on object graph features Generate object attribute features .

2. The video pre-training method based on graph generation as described in claim 1, characterized in that, In step 1, the object segmentation of the observed image is performed to obtain the object vertices. for: For time steps in the task process Use a segmentation model to predict the currently observed image The object mask is used to uniformly sample within the predicted object mask to obtain the object vertices. .

3. The video pre-training method based on graph generation as described in claim 1, characterized in that, In step 1, key interaction point estimation is performed to obtain visual action vertices. For: From the observed image Extract the fingertip position as the key interaction point to obtain the visual action vertex. .

4. The video pre-training method based on graph generation as described in claim 1, characterized in that, The object graph feature-based Generate object attribute features for: Object graph features The hidden embeddings are input into the Long Short-Term Memory (LSTM) module and output. Mapped to object attribute features .

5. The video pre-training method based on graph generation as described in claim 1, characterized in that, The graph generation modeling in step 2 further includes multi-level graph modeling, specifically including: In the figure Introducing a global vertex layer in the graph: medium object The vertices of the objects are leaf vertices. K-means clustering is used to divide them into non-overlapping clusters based on the spatial positions of the vertices. A root global vertex is added to each cluster, and its spatial position is the average of the spatial positions of all leaf vertices in that cluster. All objects The root global vertex constitutes the global vertex layer; Construct edges to connect the hierarchical structure: pairs of directed edges between all root global vertices and pairs of directed edges from each leaf vertex to the corresponding root global vertex; The root global vertex and leaf vertices belonging to the same object are assigned the same semantic category and object attributes, forming a multi-level graph; Step 2 will add the graph containing the object's attributes. Input the first graph into a neural network to generate vertex features. for: The past H The location information, semantic category attributes, and estimated object attributes at each time step are integrated and mapped into a second vertex embedding. This embedding is then aggregated along the edges of the multi-level graph using a first graph neural network to obtain vertex features. .

6. The video pre-training method based on graph generation as described in claim 1, characterized in that, In step 3, an image generator is used to process the observed image. Image encoding Language instruction encoding and the vertex features Integrate and generate graph features And thus generate the future Step-by-step graph sequence for: BERT is used to encode the language text to obtain the language instruction encoding. ; The observed image is encoded using a ResNet network to obtain the image encoding. ; Image encoding using transformer Language instruction encoding and the vertex features Integrate to obtain graph features ; Graphic features The input is fed into the prediction head of a two-layer multilayer perceptron (MLP) to generate future graph sequences.

7. The video pre-training method based on graph generation as described in claim 1 or 5, characterized in that, The mapping uses MLP.

8. An action prediction method based on graph generation, characterized in that, include: Graph-guided strategy training steps: Use a demonstration dataset of agents with action labels. Using a graph generation model obtained through any one of the video pre-training methods of claims 1-7, the observed image from the agent's perspective is predicted. Future graph sequence ; using future diagram sequences Observe the image and agent state As input, action labels serve as the ideal output, and the training graph guides the strategy. Prediction step: The agent acquires the currently observed image. and agent state The graph generation model is used to generate the future. Step-by-step graph sequence ; with the generated graph sequence Current observation image and agent state As input, a pre-trained graph-guided strategy is used to predict a series of actions of the agent. .

9. The motion prediction method as described in claim 8, characterized in that, In the graph-guided strategy training step, future graph sequences are used. Observe the image and agent state As input, action labels are the ideal output; the training graph guidance strategy is as follows: A state encoder composed of transformer blocks is used to process the graph sequence at time step t. Observe the image and agent state Encode to obtain the corresponding projection mark. The state representation is generated using a transformer. ; Representing the state With the projection marker of the graph sequence Connect to obtain policy conditions ; Using MLP action head based Generate action sequences; The loss is calculated using the generated action sequence and action labels. The state encoder and MLP action head in the graph-guided policy are then trained via backpropagation to complete the training of the graph-guided policy.

Images