Video generation model training method, video generation method, device, and storage medium

Abstract

The application discloses a video generation model training method, a video generation method, equipment and a storage medium. The method comprises: performing full-parameter fine-tuning on a pre-trained video generation model based on a first operation data set; the first operation data set comprises a plurality of operation task samples, each operation task sample comprising a text instruction description and a corresponding operation video frame sequence; performing low-rank adaptive fine-tuning on the video generation model after full-parameter fine-tuning based on a second operation data set to obtain a video generation model adapted to a target scene; and the second operation data set comprises an operation task video frame sequence of a target robot in a target scene and a corresponding text instruction. In this way, high-fidelity and scalable robot operation data can be generated at low cost and high efficiency, thereby improving the generalization capability, efficiency and task migration performance of the robot model in diversified and real scenes.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a video generation model training method, a video generation method, a computing device, and a readable storage medium. Background Technology

[0002] With the rapid development of embodied intelligence and robot learning, robots increasingly rely on large-scale, high-quality operational demonstration data for training to perform complex tasks (such as grasping, placing, and assembling). Currently, the main methods for acquiring such data include acquiring real-machine data based on human teleoperation and synthesizing simulation data based on physical simulation environments. However, acquiring real-machine data based on human teleoperation suffers from low efficiency and high cost, making it difficult to support the learning needs of large-scale, multi-task scenarios. On the other hand, synthesizing simulation data based on physical simulation environments differs significantly from the real environment in terms of visual appearance and physical interaction, leading to a decline in the generalization performance of models trained on it in real-world scenarios. Therefore, how to improve the generalization ability and efficiency of robot models in diverse and real-world scenarios has been a continuous research focus. Summary of the Invention

[0003] The purpose of this application is to provide a video generation model training method, a video generation method, a computing device, and a readable storage medium, which can generate high-fidelity and scalable robot operation data at low cost and high efficiency, thereby improving the robot model's generalization ability, efficiency, and task transfer performance in diverse and real-world scenarios.

[0004] To achieve the above objectives: In a first aspect, embodiments of this application provide a video generation model training method, characterized in that the method includes: The pre-trained video generation model is fine-tuned with all parameters based on the first operation dataset; the first operation dataset includes multiple operation task samples, each operation task sample includes a text instruction description and a corresponding operation video frame sequence. Based on the second operation dataset, the video generation model after full parameter fine-tuning is subjected to low-rank adaptation fine-tuning to obtain the video generation model adapted to the target scene; the second operation dataset includes video frame sequences of robot operation tasks in the target scene and corresponding text instructions.

[0005] In some embodiments, the full parameter fine-tuning is performed using a course learning strategy to dynamically adjust the sample weights based on the difficulty of the operation task samples during training iterations.

[0006] In some embodiments, in each training iteration of the full parameter fine-tuning, the following steps are performed to dynamically determine sample weights: Based on the parameter state of the video generation model in the current training iteration, calculate at least one difficulty evaluation index for each operation task sample in the current batch of operation task samples. Based on the at least one difficulty assessment metric, determine the dynamic loss weight for the current training iteration of each of the operation task samples; The dynamic loss weights are used to calculate the weighted loss of the current batch of operation task samples, and the model parameters are updated.

[0007] In some embodiments, determining the dynamic loss weight for the current training iteration of each of the operation task samples based on the at least one difficulty assessment metric includes: A weighted summation is performed on at least one difficulty assessment index of the target operation task sample to determine the comprehensive difficulty index of the target operation task sample; the target operation task sample is any operation task sample in the current batch. Based on the comprehensive difficulty index and the progress of the current training iteration, the dynamic loss weight of the target operation task sample is calculated; wherein, the dynamic loss weight is negatively correlated with the comprehensive difficulty index, and the strength of the negative correlation weakens as the training progress increases.

[0008] In some embodiments, the difficulty assessment metrics include at least one of the following; The video generation model has uncertainty in predicting the operation task samples; The complexity of inter-frame changes in the video frame sequence of the operation task sample; The text instructions in the operation task sample describe the complexity of the actions. The semantic inconsistency between the video content and text instructions in the operation task sample.

[0009] In some embodiments, the inter-frame variation complexity is measured based on the error of the pre-trained video generation model in predicting noise in the sequence of operation video frames of the operation task sample under initial parameter conditions. The inter-frame variation complexity is measured by calculating the average norm of the differences between consecutive video frames based on the frame-level features of the consecutive video frames. The complexity of the action is measured based on the number of verbs identified after syntactic analysis of the text instruction description. The semantic inconsistency is measured based on the similarity between video features and text features.

[0010] Secondly, embodiments of this application provide a video generation method, the method comprising: Acquire at least one target frame and corresponding operation text instruction from a robot operation video clip; The target frame is subjected to controllable image editing in at least one dimension to obtain the edited target frame; the dimension includes at least one of the following: object replacement, position adjustment, quantity change, background replacement, and lighting adjustment; The edited target frame and the corresponding operation text instruction are input into the video generation model to obtain the robot operation video sequence output by the video generation model; the video generation model is trained based on the video generation model training method described in the first aspect.

[0011] In some embodiments, the method further includes: In response to performing object replacement editing on the target frame, the corresponding object description in the operation text instruction is modified synchronously.

[0012] Thirdly, embodiments of this application provide a computing device, including: a storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the video generation model training method as described in the first aspect and / or the video generation method as described in the second aspect.

[0013] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements the video generation model training method as described in the first aspect and / or the video generation method as described in the second aspect.

[0014] The video generation model training method, video generation method, computing device, and readable storage medium provided in this application embodiment include: fine-tuning a pre-trained video generation model with full parameters based on a first operation dataset. The first operation dataset includes multiple operation task samples, each operation task sample including a text instruction description and a corresponding operation video frame sequence; fine-tuning the fully parameter-tuned video generation model with low-rank adaptation based on a second operation dataset to obtain a video generation model adapted to the target scene; the second operation dataset includes a sequence of operation task video frames of the robot in the target scene and corresponding text instructions. Thus, by learning from the first and second operation datasets sequentially, the visual content of the video generated by the trained video generation model originates from learning from real-world operation videos, preserving realistic lighting, textures, physical interaction rationality, and robot motion patterns. Simultaneously, the video generated by this model is highly correlated with the target scene and can be directly used as training data for robot imitation learning or reinforcement learning in this specific application scenario, resulting in high data utility. Therefore, the above method achieves efficient acquisition of a model capable of generating high-fidelity, highly relevant scene-specific operation videos without relying on massive amounts of real robot data. Using this model, high-fidelity, scalable robot operation data can be generated at low cost and high efficiency, thereby improving the robot model's generalization ability, efficiency, and task transfer performance in diverse and real-world scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the video generation model training method provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the implementation process of the video generation model training method provided in an embodiment of the present invention.

[0017] Figure 3 This is a flowchart illustrating the video generation method provided in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0020] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0021] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0022] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0023] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0024] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0025] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0026] With the rapid development of embodied intelligence and robot learning, robots increasingly rely on large-scale, high-quality operational demonstration data for training to perform complex tasks (such as grasping, placing, and assembling). Currently, the main methods for acquiring such data include acquiring real-machine data based on human teleoperation and synthesizing simulation data based on physical simulation environments. The process of acquiring real-machine data based on human teleoperation is as follows: the operator controls the robot to perform tasks such as grasping, carrying, connecting, and assembling in a real environment through a force feedback handle, virtual reality controller, or remote control interface, and simultaneously records video of the robot's operation process and related sensor data through a first-person view camera or multi-view camera. While this method can obtain high-quality, accurate, and semantically clear expert-level operational demonstration videos, it suffers from fundamental drawbacks such as low acquisition efficiency, high labor costs, and limited coverage of scenarios and task types, making it difficult to support the learning needs of large-scale, multi-task applications. The process of synthesizing simulation data based on a physical simulation environment is as follows: Virtual scenes and robot models are constructed in simulation platforms such as MuJoCo and Isaac Sim. Various operational tasks are automatically executed within the simulation environment using scripts or reinforcement learning algorithms, generating a large number of data samples containing motion trajectories, video frame sequences, and state information. While this method can quickly generate massive amounts of data at a low cost and allows for flexible control of variables, simulation data often differs significantly from the real environment. Issues include unrealistic visual appearance, inaccurate physical interactions, and inconsistencies between robot motion trajectories and actual robotic arm dynamics. This leads to a significant performance degradation of models trained on simulation data when transferred to real-world environments. Therefore, the industry urgently needs a new method that can generate high-fidelity robot operation data at low cost and high efficiency to bridge the gap between the scale of real-world data and the quality of simulation data.

[0027] To address the aforementioned problems, this application provides an adaptation training method for a video generation model. This method can be executed by an adaptation training device for a video generation model provided in this application. This device can be implemented in software and / or hardware, such as a server or computer. In this embodiment, a computing device is used as the execution subject of the method. (See reference...) Figure 1 The method provided in this embodiment includes: S101, Fine-tune the pre-trained video generation model with all parameters based on the first operation dataset; the first operation dataset includes multiple operation task samples, each operation task sample includes a text instruction description and a corresponding operation video frame sequence.

[0028] In this context, an operation task sample refers to a complete data unit, including text instructions describing an operation task and a corresponding sequence of video frames demonstrating the task execution process. A pre-trained video generation model is a model pre-trained on a large-scale general-purpose video dataset, capable of generating videos from images and text, such as models based on the diffusion Transformer architecture. A pre-trained video generation model can be understood as an existing video generation model. Full parameter fine-tuning refers to updating all parameters of the pre-trained video generation model during transfer learning to adapt it to the new task. Performing full parameter fine-tuning on the pre-trained video generation model based on the first operation dataset aims to transfer the capabilities of the general-purpose video generation model to the domain of robot operation video generation.

[0029] The first operation dataset contains a large number of text-video pairs, with the video content representing the operation process from a first-person perspective. This dataset can be derived from publicly available human operation datasets (such as Ego4D, EPIC-KITCHENS) and / or robot operation datasets (such as RT-1, BridgeData, etc.). In this embodiment, the existing LTX-Video model can be used as a pre-trained video generation model, with the initial frame as its input. and text instructions The output is a video sequence. .

[0030] In one embodiment, full parameter fine-tuning is performed using a course learning strategy to dynamically adjust sample weights based on the difficulty of the task samples during training iterations.

[0031] In this embodiment, the curriculum learning strategy serves as a training strategy. By allowing the model to learn from simple samples first and gradually transition to more complex ones, it improves training efficiency and model performance. Specifically, in this embodiment, the curriculum learning strategy adjusts the weight of each task sample in the loss function based on the dynamically calculated difficulty of the task sample. This allows the model to dynamically adjust its learning focus according to the sample difficulty, effectively avoiding instability or slow convergence caused by high-difficulty samples in the early stages of training, thus improving training efficiency.

[0032] In one embodiment, in each training iteration of full parameter fine-tuning, the following steps are performed to dynamically determine sample weights: Based on the parameter state of the video generation model in the current training iteration, calculate at least one difficulty assessment index for each operation task sample in the current batch of operation task samples. Based on at least one difficulty assessment metric, determine the dynamic loss weight for the current training iteration of each operation task sample; The model parameters are updated by using dynamic loss weights to calculate the weighted loss of the current batch of operation task samples.

[0033] In each training iteration, several operation task samples are sampled from the first operation dataset as the operation task samples for the current batch. During the current training iteration, for each operation task sample in the current batch, at least one difficulty evaluation metric can be calculated. These difficulty evaluation metrics can reflect the learning difficulty of the model for that operation task sample from different dimensions. After calculating at least one difficulty evaluation metric for each operation task sample, the dynamic loss weight for each operation task sample in the current training iteration can be determined by weighted summation or other methods. Then, the dynamic loss weight is used to calculate the weighted loss of the operation task samples in the current batch, obtaining the total loss of all operation task samples in the current batch. Combining the total loss of all operation task samples in the current batch, the gradient is calculated using the backpropagation algorithm, and the model parameters are adjusted using an optimizer such as Adam.

[0034] Here, we assume that the loss for each task sample during training is denoted as L. i Then, we can define the total loss L corresponding to the current batch of operation task samples. total for: w i (t) represents the sample of the operation task. i The dynamic loss weight.

[0035] In one embodiment, the dynamic loss weight for the current training iteration of each task sample is determined based on at least one difficulty assessment metric, including: The comprehensive difficulty index of the target operation task sample is determined by weighted summation of at least one difficulty assessment index for the target operation task sample; the target operation task sample is any operation task sample in the current batch. Based on the comprehensive difficulty index and the progress of the current training iteration, the dynamic loss weight of the target operation task sample is calculated; among which, the dynamic loss weight is negatively correlated with the comprehensive difficulty index, and the strength of the negative correlation weakens as the training progress increases.

[0036] Each difficulty assessment indicator corresponds to a preset weight coefficient, which can be set based on the importance of the indicator. After obtaining at least one difficulty assessment indicator for the target operation task sample, the computing device can perform a weighted summation of the at least one difficulty assessment indicator for the target operation task sample according to the preset weight coefficients for each indicator, and use the weighted summation result as the comprehensive difficulty indicator of the target operation task sample. This comprehensive difficulty indicator can be a numerical value. After obtaining the comprehensive difficulty indicator of the target operation task sample, the dynamic loss weight of the target operation task sample can be calculated in conjunction with the progress of the current training iteration. The operation task sample... i Dynamic loss weight w i (t) can be derived from the formula Perform the calculation, D i Represents operation task sample i The overall difficulty index, where t represents the current training steps, and α(t) is the course progress control parameter, set as... T max This represents the total number of training steps. The learning mechanism of this course can achieve the following goals: in the early stages of training, α(t) is large, and the model prioritizes learning low-difficulty samples; in the later stages of training, α(t) is small, and the model gradually learns high-difficulty samples, with the weights of each sample tending to be balanced in the final stage.

[0037] In one embodiment, the difficulty assessment index includes at least one of the following; Uncertainty in the prediction of operational task samples by the video generation model; The complexity of inter-frame changes in the video frame sequence of the operation task sample; The complexity of actions described in the text instructions of the task sample; The degree of semantic inconsistency between the video content and text instructions in the operation task sample.

[0038] Prediction uncertainty is measured by the error of a pre-trained video generation model in predicting noise from a video frame sequence of a task sample under its initial parameter state. A larger error indicates greater uncertainty on the part of the model and higher learning difficulty. Prediction uncertainty can also be called model uncertainty, and it measures the model's understanding of the video sequence v in the sample. i The difficulty of mastering the model is defined as the prediction noise error of the model across the entire video: ,in true (v i ) represents the actual noise in the entire video. pred (v iThe noise is the noise predicted by the model. Here, the noise is calculated by a pre-trained video generation model, such as an untuned LTX-Video model. Similarly, the calculated prediction uncertainty score needs to be normalized, i.e. m U and M U These are the minimum and maximum values ​​of the prediction noise error for all samples in the training set.

[0039] The complexity of inter-frame variation can be measured by calculating the average norm of the differences between frame-level features extracted from consecutive video frames by a pre-trained visual model. A larger average value indicates more intense object movement or interaction in the video, more complex visual dynamics, and higher learning difficulty. The complexity of inter-frame variation, C, is... i It can be used to measure the intensity of visual dynamic changes in a video, reflecting the degree of change in object displacement, user interaction, etc. during the operation process. Specifically, it can be defined as the average value of the inter-frame feature differences: F i,t Let C represent the frame-level features of the i-th sample in frame t, which can be specifically calculated using the ViT-B / 16 model. Next, the calculated C... i Then perform a normalization operation: m c and M C These are the minimum and maximum values ​​of this value on the training set, respectively. C i A larger value indicates a greater degree of variation between frames, thus reflecting a higher learning difficulty for that sample. It should be noted that the training set mentioned above can be understood as the set of operation task samples for the current batch.

[0040] Action complexity is measured by the number of verbs identified after syntactic analysis of the text instruction description. A higher number of verbs generally indicates more complex operation steps and a higher learning difficulty. The action complexity T... i It can be used to measure the number of actions from the perspective of text instructions, that is, to count the number of verbs contained in text instructions, specifically defined as... Among them, verb_count(t i The number of verbs in the text instruction is represented by ) (which can be extracted using a dependency parser), m T and M T These are the minimum and maximum values ​​of this value on the current batch of operation task samples, used for normalization calculations.

[0041] Semantic inconsistency can be measured by calculating the similarity between video and text features using a pre-trained cross-modal model. Lower similarity indicates a poorer match between the video content and the text instructions, suggesting the sample may be more ambiguous or noisy, and thus more difficult to learn. Specifically, a pre-trained cross-modal model (such as VideoCLIP) can be used to calculate the similarity between text and video features of the task samples. s i : And define semantic inconsistency as: Samples with low semantic similarity (e.g., ambiguous instructions) will receive higher difficulty scores.

[0042] Taking the comprehensive difficulty index as an example, for each operation task sample (v) i , t i ), can be calculated according to the formula Calculate its overall difficulty score D i C i For the complexity of inter-frame variation, T i For action complexity, U i To predict uncertainty, S i λ represents the semantic inconsistency degree. C , λ T , λ U , λ S These are the weighting coefficients for each item. The overall difficulty score D is calculated for each task sample. i The training process, from easy to difficult, is controlled by using dynamic weighted loss.

[0043] S102, Based on the second operation dataset, perform low-rank adaptation fine-tuning on the video generation model after full parameter fine-tuning to obtain a video generation model adapted to the target scene; the second operation dataset includes video frame sequences of robot operation tasks in the target scene and corresponding text instructions.

[0044] Here, "robot" refers to a robot that needs to be trained using operation video sequences generated by a video generation model, and "target scenario" refers to a real-world work scenario in which the robot performs its operation tasks. Low-rank adaptation fine-tuning is a parameter-efficient fine-tuning technique that injects trainable low-rank matrices into the original linear layers of the model, keeping most of the original model's parameters unchanged to adapt to the new task.

[0045] The operation task samples in the second operation dataset can be obtained through expert teleoperation demonstrations when a robot (e.g., our company's six-axis collaborative robotic arm, model X) performs operation tasks in its actual deployment target work environment (e.g., our company's product assembly line). Each operation task sample in the second operation dataset contains a first-person perspective video of a complete operation task, along with manually annotated text instructions that precisely describe the operation task. For example, the operation task video frame sequence in the second operation dataset records the complete visual process of the robot performing a specific operation task (such as grasping, placing, or assembling) in the target scene. The video is usually captured from a first-person perspective (e.g., a camera mounted at the end of the robotic arm), and the corresponding text instructions precisely correspond to the video content and describe the operation task (e.g., "Use the robotic arm to pick up the red cube and put it in the box on the right").

[0046] While the second operational dataset is typically smaller (e.g., tens to hundreds of samples) and smaller than the first operational dataset, it possesses extremely high specificity and fidelity. By performing low-rank adaptation fine-tuning on the second operational dataset, the video generation model can quickly and efficiently transfer the general operational knowledge learned through full-parameter fine-tuning to specific robot models and operational scenarios, achieving accurate scene adaptation. When training the model using low-rank adaptation fine-tuning, trainable low-rank matrices can be added next to the linear layers of the model. Only these newly added parameters are updated, thus achieving accurate alignment of the model in specific visual appearances, robot motion styles, and scene semantics with a minimal number of parameters (typically less than 1% of the original model), while avoiding overfitting due to the small amount of data.

[0047] In this embodiment, by fine-tuning the pre-trained video generation model with all parameters based on the first operation dataset, the model can gain a deep understanding of the cross-modal relationship between textual instructions (such as "grab" and "place") and visual dynamic temporal sequences (such as the trajectory of the grasping action and the displacement of the object) in the operation task. This enables the model to acquire the basic ability to generate logically consistent general operation videos. Then, by performing low-rank adaptation fine-tuning on the fully parameter-fine-tuned video generation model based on the second operation dataset, the output video can be made consistent with the second operation dataset in terms of visual appearance (such as the model and color of the target robot and the desktop texture of the target scene) and motion characteristics (such as the kinematic and dynamic characteristics unique to the target robotic arm). In other words, by learning from the first and second operation datasets in sequence, the visual content of the video generated by the trained video generation model originates from learning from real-world operation videos. It can retain realistic lighting, texture, physical interaction rationality, and robot motion laws. At the same time, the video generated by this model is highly correlated with the target scene and can be directly used as training data for robot imitation learning or reinforcement learning in this specific application scenario, resulting in high data utility. Therefore, the above method achieves efficient acquisition of a model capable of generating high-fidelity, highly relevant scene-specific operation videos without relying on massive amounts of real robot data. Using this model, high-fidelity, scalable robot operation data can be generated at low cost and high efficiency, thereby improving the robot model's generalization ability, efficiency, and task transfer performance in diverse and real-world scenarios.

[0048] For example, see Figure 2 The training process of the aforementioned video generation model can be divided into two stages: a domain-specific adaptation fine-tuning stage and an enterprise-specific scenario fine-tuning stage. Through progressive training in these two stages, the general video generation model can be efficiently transferred to the robot operation domain, thereby generating high-quality, controllable robot operation video data to enrich the enterprise's internal robot vision and imitation learning dataset.

[0049] Phase 1: Fine-tuning training for operational domain adaptation In the first phase, publicly available human and robot first-person perspective operation datasets from the internet are used as training data sources to adapt the general video generation model to the "operation task" video generation direction. Human operation datasets can include Ego4D, EgoVid-5M, EPIC-KITCHENS, Kinetics-700, SSV2, HowTo100M, etc. Robot operation datasets can include RT-1, Bridge, DROID, Open X-Embodiment, etc. Regarding the model structure, the open-source LTX-Video-13B can be used as the general video generation model. This model is a large-scale general video generation network composed of a variational autoencoder and a diffusion transformer. The input includes an initial image frame and a text instruction description, and the output is a sequence of generated video frames.

[0050] In this stage, the model undergoes full-parameter fine-tuning to fully adapt to the semantic style, temporal features, and action logic of the "operation videos." Since the operation video data used in the first stage comes from a wide range of sources, including different environments, different robot bodies, and different task types, the various samples differ significantly in visual dynamic complexity, action length, and semantic consistency. If all samples are directly used for training with the same weight, the model is prone to training instability and difficulty in convergence, and the generated videos are of uneven quality, often exhibiting problems such as disjointed actions or semantic shifts. Therefore, a curriculum learning mechanism oriented towards robot operation is introduced in the first stage of training to implement a gradual training strategy from easy to difficult, thereby improving the model's stability and generation performance.

[0051] To measure the learning difficulty of different video samples, a difficulty evaluation method based on multi-dimensional features was designed. For each sample (v) in the training set... i , t i ), calculate its overall difficulty score D i The definition is as follows: C i For the complexity of video frame changes, T i For action complexity, U i For model uncertainty, S i λ represents the semantic inconsistency degree. C , λ T , λ U , λ S These are the weighting coefficients for each item. Then, the overall difficulty score D is calculated based on each sample. i The training process, from easy to difficult, is controlled by using dynamic weighted loss.

[0052] Phase Two: Fine-tuning training for enterprise-specific scenarios In the second phase, scenario-level transfer fine-tuning was performed for specific robot models and operating environments within the enterprise. The training data for this phase came from first-person expert demonstration videos collected when the enterprise's own robots performed tasks in real-world scenarios. This video data was collected remotely via human intervention and includes visual information and textual instruction annotations from the actual robots performing tasks. Due to the limited amount of data and high acquisition costs in the enterprise's own scenarios, a low-rank adaptation fine-tuning strategy was adopted in this phase. This method achieves the following technical effects by inserting low-rank matrices into the linear layers of the model for parameter adaptation: avoiding the overfitting risk associated with full parameter updates; reducing memory and computational resource consumption; maintaining the basic capabilities of the original model; and improving the model's adaptation speed for specific enterprise tasks (such as grasping, placing, and assembling with a specific type of robotic arm).

[0053] The input and output formats in this stage are the same as in the first stage, but the training objectives are more focused on scene transfer and action style alignment. Specifically, at the visual level, the model is able to generate videos that are consistent with the visual characteristics (color, viewpoint, ambient lighting) of the enterprise robot; at the behavioral level, the action style of the generated videos is consistent with the dynamic laws of the actual actions performed by the enterprise robot; and at the semantic level, the generated results are more in line with the operational intent in the enterprise task definition.

[0054] Through low-rank adaptation fine-tuning in this stage, the model's parameters are only slightly updated, but the generated results can be smoothly migrated from "general operation videos" to "operation videos in specific scenarios", thereby achieving efficient scenario adaptation of the model.

[0055] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a video generation method. This method can be executed by a video generation device provided in this application embodiment. This device can be implemented in software and / or hardware, such as a server or computer. In this embodiment, a computing device is taken as the executing entity of the method. Figure 3 As shown, the method provided in this embodiment includes: S201, acquire at least one target frame of a robot operation video segment and the corresponding operation text instruction.

[0056] The target frame refers to a single frame image extracted from a real video clip of robot operation, used as the starting point for video generation; it is usually the initial frame of the video. Optionally, if the initial frame of the video is the same as or does not change significantly from the second and / or third frame, the second or third frame can be used as the target frame.

[0057] S202, perform controllable image editing on the target frame in at least one dimension to obtain the edited target frame; the dimension includes at least one of the following: object replacement, position adjustment, quantity change, background replacement, and lighting adjustment.

[0058] Controlled image editing refers to the process of using an image generation model to modify specific attributes (such as objects, background, lighting, etc.) of an input image according to specified editing instructions (such as text descriptions). This involves extracting the target frame (usually the first frame) and its corresponding operation text instructions from an existing library of real robot operation videos. Subsequently, to increase data diversity, large image editing models (such as Qwen-Image) can be used on the target frame. Controllable editing is performed in at least one dimension to generate the edited target frame. The dimensions can include at least one of the following: object replacement, position adjustment, quantity change, background replacement, and lighting adjustment. Object replacement refers to changing the object category, such as replacing a cup with a bottle; position adjustment refers to adjusting the object's position and posture; quantity change refers to changing the number of objects, such as changing from one to multiple; background replacement refers to changing the operating background, such as changing from a laboratory table to a factory table; and lighting adjustment refers to adjusting the direction, intensity, or color style of lighting to simulate different environments.

[0059] S203, input the edited target frame and the corresponding operation text instructions into the video generation model to obtain the robot operation video sequence output by the video generation model.

[0060] The video generation model is trained based on the video generation model training method provided in the aforementioned embodiments.

[0061] The edited target frame and its corresponding operation text instructions are input into a trained video generation model, enabling the model to automatically reason and generate a complete robot operation video sequence that executes instructions starting from the target frame. This approach aims to expand upon a limited collection of real-world robot operation videos, generating a large amount of diverse new data. The generated operation video sequences, combined with the original real-world video data, form an enhanced training dataset for training downstream robot vision or policy models.

[0062] In summary, the video generation method provided in the above embodiments can quickly generate augmented data for training robots through the video generation model, which significantly reduces the human and time costs of data collection, and increases the data scale while maintaining the diversity of operations. At the same time, the generated data not only retains the lighting, texture and background information in the real environment, but also accurately reflects the robot's operation actions and object interaction characteristics, thereby improving the training effect and generalization ability of the model in diverse and real-world scenarios.

[0063] In one embodiment, the method further includes: In response to performing object replacement editing on the target frame, the corresponding object description in the operation text instruction is modified synchronously.

[0064] If it is necessary to replace an object in the target frame, an object replacement editing command can be input into the computing device. The computing device can respond by performing object replacement editing on the target frame and simultaneously modifying the corresponding object description in the operation text command to maintain consistency between the image and text. For example, assuming the robot video operation segment is a video of a robot picking up a cup, if the cup in the initial frame is replaced with a bottle, then the word "cup" in the operation text command needs to be modified to "bottle".

[0065] In this way, when editing object replacements, the text instructions are updated synchronously, ensuring a high degree of consistency between the text and images. This avoids the problem of generating chaotic videos due to mismatch between text and images, and guarantees the overall quality and usability of the augmented data.

[0066] For example, after the video generation model completes the training of the above two stages, the model has the ability to generate high-quality robot operation videos based on the initial frame and text instructions. To achieve data augmentation, the inference stage is designed with the following workflow: 1) Initial frame extraction: Select the initial frame of each video segment from a small number of actual robot operation videos collected by the company itself. 0 is used as the starting point for generation. The text instructions or operation task descriptions corresponding to each initial frame are saved. T Used as input for subsequent video generation conditions.

[0067] 2) Initial Frame Image Editing: To enrich the diversity and scene coverage of the generated data, the initial frame is edited. 0. Controllable editing at the image level. Specifically, Qwen-Image image editing can be used to edit large models, enabling the following controlled modifications: Object Category: Replace the object to be manipulated in the initial frame, for example, replace the cup with a bottle, box, or other tool. (In this case, the corresponding text instructions also need to be modified because the object to be manipulated has changed.) Object position and orientation: Adjust the position and orientation of objects in the image, such as moving an object from the left side of the table to the right side, or rotating it to a specific angle.

[0068] Number of objects: Increase or decrease the number of objects in the scene, for example, increase from a single cup to three cups, or remove some objects to create a sparse scene.

[0069] Background: Replace or modify the scene background, for example, changing it from a laboratory desktop to a factory desktop or kitchen countertop. This allows for background changes while keeping the initial objects unchanged, enhancing data diversity.

[0070] Lighting and Color Style: Adjust the direction, brightness, or overall color style of scene lighting to simulate different shooting environments. This can enhance the model's generalization ability under different visual conditions.

[0071] 3) Initial frame-based video generation: The edited initial frame and the corresponding text instruction T are input into the video generation model, which has undergone two-stage fine-tuning training. The model will then generate a complete video sequence. V’ ={ I 1', I 2', … , I n The generated new video data can expand the size of the original operational dataset and enrich the diversity of the data.

[0072] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 4 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 4 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 4 The memory 311 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, it implements a video generation model training method and / or a video generation method applied to the above-mentioned computing device.

[0073] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4The general designated all buses as Bus System 313.

[0074] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0075] The memory 311 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0076] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video generation model training method and / or video generation method as described above.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for training a video generation model, characterized in that, The method includes: The pre-trained video generation model is fine-tuned with all parameters based on the first operation dataset; the first operation dataset includes multiple operation task samples, each operation task sample includes a text instruction description and a corresponding operation video frame sequence. Based on the second operation dataset, the video generation model after full parameter fine-tuning is subjected to low-rank adaptation fine-tuning to obtain the video generation model adapted to the target scene; the second operation dataset includes video frame sequences of robot operation tasks in the target scene and corresponding text instructions.

2. The method according to claim 1, characterized in that, The full parameter fine-tuning adopts a course learning strategy to dynamically adjust the sample weights according to the difficulty of the operation task samples during the training iteration process.

3. The method according to claim 2, characterized in that, In each training iteration of the full parameter fine-tuning, the following steps are performed to dynamically determine the sample weights: Based on the parameter state of the video generation model in the current training iteration, calculate at least one difficulty evaluation index for each operation task sample in the current batch of operation task samples. Based on the at least one difficulty assessment metric, determine the dynamic loss weight for the current training iteration of each of the operation task samples; The dynamic loss weights are used to calculate the weighted loss of the current batch of operation task samples, and the model parameters are updated.

4. The method according to claim 3, characterized in that, The step of determining the dynamic loss weight for the current training iteration of each operation task sample based on the at least one difficulty assessment metric includes: A weighted summation is performed on at least one difficulty assessment index of the target operation task sample to determine the comprehensive difficulty index of the target operation task sample; the target operation task sample is any operation task sample in the current batch. Based on the comprehensive difficulty index and the progress of the current training iteration, the dynamic loss weight of the target operation task sample is calculated; wherein, the dynamic loss weight is negatively correlated with the comprehensive difficulty index, and the strength of the negative correlation weakens as the training progress increases.

5. The method according to claim 3 or 4, characterized in that, The difficulty assessment indicators include at least one of the following: The video generation model has uncertainty in predicting the operation task samples; The complexity of inter-frame changes in the video frame sequence of the operation task sample; The text instructions in the operation task sample describe the complexity of the actions. The semantic inconsistency between the video content and text instructions in the operation task sample.

6. The method according to claim 5, characterized in that, The prediction uncertainty is measured by the error of the pre-trained video generation model in predicting noise in the sequence of operation video frames of the operation task sample under the initial parameter state. The inter-frame variation complexity is measured by calculating the average norm of the differences between consecutive video frames based on the frame-level features of the consecutive video frames. The complexity of the action is measured based on the number of verbs identified after syntactic analysis of the text instruction description. The semantic inconsistency is measured based on the similarity between video features and text features.

7. A video generation method, characterized in that, The method includes: Acquire at least one target frame and corresponding operation text instruction from a robot operation video clip; The target frame is subjected to controllable image editing in at least one dimension to obtain the edited target frame; the dimension includes at least one of the following: object replacement, position adjustment, quantity change, background replacement, and lighting adjustment; The edited target frame and the corresponding operation text instruction are input into the video generation model to obtain the robot operation video sequence output by the video generation model; the video generation model is trained based on the video generation model training method according to any one of claims 1 to 6.

8. The method according to claim 7, characterized in that, The method further includes: In response to performing object replacement editing on the target frame, the corresponding object description in the operation text instruction is modified synchronously.

9. A computing device, characterized in that, include: A storage medium and a controller, wherein a computer program is stored on the storage medium, and when executed by the controller, the computer program implements the video generation model training method as described in any one of claims 1 to 6 and / or the video generation method as described in any one of claims 7 to 8.

10. A readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the video generation model training method as described in any one of claims 1 to 6 and / or the video generation method as described in any one of claims 7 to 8.

Images