Robot arm control method based on video diffusion model and related equipment

Abstract

The invention provides a robotic arm control method based on a video diffusion model and related equipment, and the method comprises the steps: extracting a plurality of sample historical image track features and sample target image track features from a robotic arm motion video, and carrying out noise diffusion to obtain sample track noise diffusion features; based on the sample historical image track features and the sample track noise diffusion features, taking noise diffusion features associated with the sample target image as a training target, and training a preset model; inputting the current image track feature of the robotic arm into a trained target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, and performing feature fusion based on the plurality of predicted noise diffusion features through a multi-layer cascaded action adapter contained in the target video diffusion model to obtain a plurality of predicted action features, the robot arm is controlled on the basis of the action control signals corresponding to the multiple predicted action characteristics, and therefore the accuracy of mechanical arm action prediction and control is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of robotics technology, and in particular to a robotic arm control method based on a video diffusion model and related equipment. Background Art

[0002] In the field of robotics, especially in the operation of robot arms, accurate prediction and execution of actions are the key to achieving automation and intelligence. With the development of machine learning technology, the robot's arm can learn to extract information from complex environments and make accurate judgments by training a large amount of data. However, the existing robot arm operation data has high labeling costs and high hardware requirements, and directly fits the existing data without considering the relationship between visual observation and action, which easily leads to low data utilization efficiency and cannot fully tap the potential value of the data. As a result, in the current situation of limited data, the accuracy and stability of traditional solutions in prediction and control are poor, and cannot meet the needs of actual applications. Summary of the invention

[0003] The embodiment of the present application provides a robotic arm control method based on a video diffusion model, which can effectively improve the data utilization rate of the model, and improve the accuracy and stability of robotic arm motion prediction and control.

[0004] To achieve the above-mentioned purpose, the first aspect of an embodiment of the present application provides a robot arm control method based on a video diffusion model, comprising: extracting multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from a preset robot arm action video; performing noise diffusion on the sample historical image trajectory features and the sample target image trajectory features to obtain sample trajectory noise diffusion features; based on the sample historical image trajectory features and the sample trajectory noise diffusion features, a pre-trained preset multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target to obtain a target video diffusion model for outputting the sample target image; inputting the current image trajectory features of the robot arm into the target video diffusion model to obtain multiple predicted noise diffusion features obtained by diffusion, and performing feature fusion based on the multiple predicted noise diffusion features through a multi-layer cascaded action adapter included in the target video diffusion model to obtain multiple predicted action features; performing action decoding on the multiple predicted action features according to the control parameters of the robot arm to obtain multiple action control signals, so as to control the robot arm based on the multiple action control signals.

[0005] In some embodiments, the extracting of multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from a preset robot arm motion video includes: performing video frame sampling division on the preset robot arm motion video according to the video frame timing to obtain multiple sample historical image frames and sample target image frames located after the multiple sample historical image frames; extracting sample historical image trajectory features from the sample historical image frames by a preset video feature extractor, and extracting sample target image trajectory features from the sample target image frames.

[0006] In some embodiments, performing noise diffusion on the sample historical image trajectory features and the sample target image trajectory features to obtain sample trajectory noise diffusion features includes: combining the sample historical image trajectory features and the sample target image trajectory features into a sample image trajectory feature; and adding preset noise data to the sample image trajectory features in sequence according to a time step order to obtain a sample trajectory noise diffusion feature.

[0007] In some embodiments, based on the sample historical image trajectory features and the sample trajectory noise diffusion features, the noise diffusion features associated with the sample target image are used as training targets, and a pre-trained preset multi-frame generation model is trained to obtain a target video diffusion model for outputting the sample target image, including: filling the sample historical image trajectory features with zero values ​​corresponding to the dimension of the sample target image, so that the filled sample historical image trajectory features are connected with the sample trajectory noise diffusion features along the channel dimension to obtain noise diffusion conditional features; based on the noise diffusion conditional features, the noise diffusion features associated with the sample target image are used as training targets, and a pre-trained preset multi-frame generation model is trained to obtain a target video diffusion model for outputting the sample target image.

[0008] In some embodiments, based on the noise diffusion condition feature, the noise diffusion feature associated with the sample target image is used as a training target to train a pre-trained preset multi-frame generation model to obtain a target video diffusion model for outputting the sample target image, including: obtaining a language instruction feature for indicating that the pre-trained preset multi-frame generation model predicts the sample target image; obtaining a diffusion time step corresponding to the noise diffusion condition feature; inputting the noise diffusion condition feature, the language instruction feature and the diffusion time step into the multi-frame generation model, taking the noise diffusion feature associated with the sample target image as a training target, training the preset multi-frame generation model to obtain a target video diffusion model for outputting the sample target image.

[0009] In some embodiments, the preset multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target, including: obtaining the predicted noise diffusion features obtained by the preset multi-frame generation model based on the diffusion of sample historical images; performing model loss calculation based on the predicted noise diffusion features and the noise diffusion features associated with the sample target image, so as to update the model parameters according to the obtained model loss.

[0010] In some embodiments, the current image trajectory features of the robotic arm are input into the target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, including: filling the current image trajectory features of the robotic arm with zero values ​​based on the corresponding dimensions of the sample target image, and connecting the filled current image trajectory features with Gaussian noise of the same dimension along the channel dimension to obtain the current visual features; inputting the current visual features and the corresponding language instruction features into the video diffusion model, and setting the diffusion time step of the video diffusion model to a preset maximum time step to obtain a plurality of predicted noise diffusion features obtained by diffusion.

[0011] In some embodiments, the multi-layer cascaded action adapter included in the target video diffusion model performs feature fusion based on the multiple predicted noise diffusion features to obtain multiple predicted action features, including: reading the predicted noise diffusion features of each layer of the video diffusion model through the multi-layer cascaded action adapter; performing feature fusion on the multiple predicted noise diffusion features based on the multi-head cross-attention query features preset in the action adapter to obtain multiple predicted action features.

[0012] In some embodiments, the action decoding of the multiple predicted action features according to the control parameters of the robot arm to obtain multiple action control signals includes: acquiring the control parameters of the robot arm and the adapter parameters of the action adapter; configuring the decoder parameters of the action decoder according to the control parameters; constructing a decoding loss function according to the model parameters, the decoder parameters and the adapter parameters; updating the decoder parameters based on the decoding loss corresponding to the decoding loss function, so that the action decoder performs action decoding on the multiple predicted action features to obtain multiple action control signals.

[0013] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application provides a robot arm control device based on a video diffusion model, comprising: a model training module, used to extract multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from a preset robot arm action video; perform noise diffusion on the sample historical image trajectory features and the sample target image trajectory features to obtain sample trajectory noise diffusion features; based on the sample historical image trajectory features and the sample trajectory noise diffusion features, a pre-trained preset multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target to obtain a target video diffusion model for outputting the sample target image; an action control module, used to input the current image trajectory features of the robot arm into the target video diffusion model to obtain multiple predicted noise diffusion features obtained by diffusion, and perform feature fusion based on the multiple predicted noise diffusion features through a multi-layer cascaded action adapter included in the target video diffusion model to obtain multiple predicted action features; perform action decoding on the multiple predicted action features according to the control parameters of the robot arm to obtain multiple action control signals, so as to control the robot arm based on the multiple action control signals.

[0014] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application provides a computer device, including: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, a robot arm control method based on a video diffusion model as described in any one of the first aspects is implemented.

[0015] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the robot arm control method based on the video diffusion model as described in any one of the fourth aspects.

[0016] The embodiment of the present application provides a robot arm control method and related equipment based on a video diffusion model. Through video frame sampling division and feature extraction, multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features can be extracted from a preset robot arm action video in turn, so as to reduce the dependence on labeled data and improve the data utilization rate of the model; further, the sample historical image trajectory features and the sample target image trajectory features are subjected to noise diffusion to obtain sample trajectory noise diffusion features, and based on the sample historical image trajectory features and the sample trajectory noise diffusion features, clear conditional guidance is provided for subsequent models to be trained, so that the model can perform noise prediction based on the sample historical image trajectory features and the noise diffusion features associated with the sample target image as the training target, train the pre-trained preset multi-frame generation model, and compare the predicted noise with the noise The diffusion feature updates the optimized model parameters, so that the model has the ability to predict the future state by predicting the noise, and then obtains the target video diffusion model for outputting the sample target image, thereby improving the prediction accuracy of the model; further, by inputting the current image trajectory features of the robot arm into the target video diffusion model, the video diffusion model can be feature trained to obtain multiple predicted noise diffusion features obtained by diffusion, and the multi-layer cascaded action adapter contained in the target video diffusion model performs feature fusion based on the multiple predicted noise diffusion features to obtain multiple predicted action features, and then performs action decoding on the multiple predicted action features according to the control parameters of the robot arm to avoid the time-consuming iterative denoising process of the video diffusion model, thereby quickly and effectively obtaining the action control signal for the robot arm control, effectively improving the data utilization of the model, and improving the accuracy and stability of the robot arm action prediction and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a robotic arm control method based on a video diffusion model provided in one embodiment of the present application; Figure 2 A flowchart for extracting image trajectory features in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Figure 3 A flow chart of obtaining noise diffusion characteristics of sample trajectories in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Figure 4 A flow chart of obtaining a target video diffusion model in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Figure 5 Another flow chart of obtaining a target video diffusion model in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Figure 6In the robot arm control method based on the video diffusion model provided in one embodiment of the present application, a schematic diagram of a target video diffusion model is obtained; Figure 7 A flowchart for obtaining predicted noise diffusion characteristics in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Figure 8 A flowchart of obtaining multiple action control signals in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Fig. 9 A schematic diagram of obtaining multiple motion control signals in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] In some embodiments, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms first, second, etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] In addition, unless otherwise clearly specified and limited, the term "connected" should be understood in a broad sense. For example, it can be a fixed connection or a movable connection, a detachable connection or a non-detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or the two can communicate with each other; it can be a direct connection or an indirect connection through an intermediate medium.

[0021] In the description of the embodiments of the present application, the description of reference terms "one embodiment / implementation", "another embodiment / implementation" or "certain embodiments / implementations", "in the above embodiments / implementations", etc. means that the specific features, structures, materials or characteristics described in combination with the implementation or example are included in at least two embodiments or implementations disclosed in the present application. In the disclosure of this application, the schematic representation of the above terms does not necessarily refer to the same embodiment or implementation. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart.

[0022] Currently, in the field of robotics, especially in the operation of robot arms, accurate prediction and execution of actions are the key to achieving automation and intelligence. With the development of machine learning technology, the robot's arm can learn to extract information from complex environments and make accurate judgments by training a large amount of data. However, the existing robot arm operation data has high labeling costs and high hardware requirements, and directly fits the existing data without considering the relationship between visual observation and action, which easily leads to low data utilization efficiency and cannot fully tap the potential value of the data. As a result, in the current situation of limited data, the traditional solution prediction and control accuracy and stability are poor, which cannot meet the actual application needs.

[0023] It can be understood that in order to overcome the limitations of the prior art, the present application provides a robot arm control method and related equipment based on a video diffusion model, including: extracting multiple sample historical image trajectory features and sample target image trajectory features from the robot arm action video, performing noise diffusion, and obtaining sample trajectory noise diffusion features; based on the sample historical image trajectory features and the sample trajectory noise diffusion features, a preset model is trained with the noise diffusion features associated with the sample target image as the training target; the current image trajectory features of the robot arm are input into the trained target video diffusion model to obtain multiple predicted noise diffusion features obtained by diffusion, and feature fusion is performed based on the multiple predicted noise diffusion features through the multi-layer cascaded action adapter contained in the target video diffusion model to obtain multiple predicted action features, so as to control the robot arm based on the action control signals corresponding to the multiple predicted action features, thereby improving the accuracy of robot arm action prediction and control.

[0024] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0025] refer to Figure 1 , Figure 1 A flowchart of a robotic arm control method based on a video diffusion model provided in one embodiment of the present application; in some embodiments, the present application provides a robotic arm control method based on a video diffusion model, including but not limited to the following steps: Step S110, extracting a plurality of sample historical image trajectory features and a plurality of sample target image trajectory features located after the plurality of sample historical image trajectory features from the preset robot arm motion video; Step S120, performing noise diffusion on the sample history image trajectory features and the sample target image trajectory features to obtain the sample trajectory noise diffusion features; Step S130, based on the sample historical image trajectory features and the sample trajectory noise diffusion features, taking the noise diffusion features associated with the sample target image as the training target, training the pre-trained preset multi-frame generation model to obtain a target video diffusion model for outputting the sample target image; Step S140, inputting the current image trajectory feature of the robot arm into the target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, and performing feature fusion based on the plurality of predicted noise diffusion features through a multi-layer cascaded action adapter included in the target video diffusion model to obtain a plurality of predicted action features; Step S150 , performing motion decoding on a plurality of predicted motion features according to control parameters of the robot arm to obtain a plurality of motion control signals, so as to control the robot arm based on the plurality of motion control signals.

[0026] In some embodiments, the sample historical image trajectory features in the above steps S110 to S150 can be understood as feature representations of a series of historical images extracted from a preset robotic arm action video, and these features are used for model training to help the model learn past action patterns; Sample target image trajectory features: can be understood as image features at a certain moment in the future that correspond to the sample historical image trajectory features. Sample target image trajectory features serve as training targets to help the model learn how to predict future images from historical images. Noise diffusion: It can be understood as adding noise to image features during training, with the goal of letting the model learn how to recover the original information from noisy data. This process helps improve the generalization and robustness of the model.

[0027] Sample trajectory noise diffusion feature: It can be understood as the sample image trajectory feature after noise diffusion processing, which is used to train the model to learn how to remove noise, so as to better predict future image trajectories; Pre-trained preset multi-frame generation model: It can be understood as a model that has been trained using a large amount of data and can generate continuous multi-frame images. In the application scenario of this application, the multi-frame generation model can specifically refer to the Open-Sora model, which can be used to generate diverse and physically realistic continuous video frames that are aligned with language instructions.

[0028] Target video diffusion model: It can be understood as a model that can predict future image features based on historical image features after training in step S130. The target video diffusion model is obtained by predicting the noise diffusion features associated with the sample target image based on the pre-trained multi-frame generation model; Multi-layer cascaded action adapter: It can be understood as a lightweight module for converting the target video diffusion model into an inverse dynamics model. By introducing a layer-by-layer adapter, the features of the standard prediction diffusion noise knowledge of each layer of the target video diffusion model can be fused through the multi-head cross attention and feedforward network with gating mechanism contained in the action adapter to generate action representation; Predicted action features: can be understood as action representations generated based on predicted noise diffusion features through the target video diffusion model and its multi-layer cascaded action adapter. The predicted action features contain relevant information about the predicted action. Action decoding: It can be understood as the process of converting predicted action features into specific robot arm control signals. This process can be completed by a diffusion-based action decoder. The decoder is responsible for decoding high-level action representations into low-level control signals that can be directly used to control the robot arm, such as the posture of the end effector and the state of the gripper.

[0029] Among them, through the above-mentioned steps S110 to S150, the embodiment of the present application can extract multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from the preset robot arm action video through video frame sampling division and feature extraction, so as to reduce the dependence on labeled data and improve the data utilization rate of the model; further, the sample historical image trajectory features and the sample target image trajectory features are subjected to noise diffusion to obtain the sample trajectory noise diffusion features, and based on the sample historical image trajectory features and the sample trajectory noise diffusion features, clear conditional guidance is provided for the subsequent model to be trained, so that the model can perform noise prediction based on the sample historical image trajectory features and the noise diffusion features associated with the sample target image as the training target, and train the pre-trained preset multi-frame generation model, and compare the predicted noise with the noise diffusion features The model parameters are updated and optimized so that the model has the ability to predict the future state by predicting noise, thereby obtaining a target video diffusion model for outputting a sample target image, thereby improving the prediction accuracy of the model; further, by inputting the current image trajectory features of the robotic arm into the target video diffusion model, the video diffusion model can be feature trained to obtain multiple predicted noise diffusion features obtained by diffusion, and feature fusion is performed based on multiple predicted noise diffusion features through the multi-layer cascaded action adapter contained in the target video diffusion model to obtain multiple predicted action features, and then action decoding is performed on the multiple predicted action features according to the control parameters of the robotic arm to avoid the time-consuming iterative denoising process of the video diffusion model, thereby quickly and effectively obtaining the action control signal for the robotic arm control, effectively improving the data utilization of the model, and improving the accuracy and stability of the robotic arm action prediction and control.

[0030] In some embodiments, it can be understood that the dual process theory in current cognitive science points out that slow and thoughtful reasoning can guide fast and direct control behavior, so the present application can design a two-stage training strategy to specifically target robot operations based on video diffusion generation models, aiming to improve the robot's understanding and reaction speed to the environment; specifically, in the first stage, that is, the dynamic perception visual stage, corresponding to steps S110 to S130, the present application can extract multiple sample historical image trajectory features and corresponding sample target image trajectory features from the preset robot arm action video, apply noise diffusion technology to these sample image trajectory features, generate sample trajectory noise diffusion features, and then use the sample historical image trajectory features and sample trajectory noise diffusion features, combined with the pre-trained multi-frame generation model, to train to generate a target video diffusion model that can accurately output the target image, wherein the goal of the present application is to train the model based on the existing historical observation data to predict and conceive possible future trajectories, and steps S110 to S130 can make full use of the multi-frame prediction capability of the video diffusion model, process the historical image trajectory features and the target image trajectory features through the noise diffusion technology, and generate the sample trajectory noise diffusion features. The model is optimized at this stage to better understand and simulate the dynamic characteristics of the environment.

[0031] Further, entering the second stage, namely the dynamic modulation action stage, corresponding to step S140 to step S150, the present application can input the current image trajectory features of the robot arm into the trained target video diffusion model to obtain multiple predicted noise diffusion features, and realize feature fusion through the built-in multi-layer cascade action adapter of the model to generate multiple predicted action features, and then, decode the predicted action features according to the control parameters of the robot arm to generate a series of action control signals to achieve precise control of the robot arm. The present application introduces a lightweight multi-layer cascade action adapter to combine the forward-looking prediction of the first stage with the fast adaptive action prediction. This adapter can convert the predicted noise diffusion features obtained by diffusion into predicted action features through feature fusion, effectively separating the abstract understanding of the world from the specific operation implementation, while ensuring a smooth transition between the two, and converting these predicted action features into actual control signals through the action decoding process based on the robot arm control parameters. The above process can not only improve the accuracy of the robot arm action prediction, but also enhance its ability to adapt to different tasks and environments, so as to accurately guide the operation of the robot arm.

[0032] In some embodiments, the present application can train a video generation model based on the video data of the robotic arm, so that the video generation model has the ability to recognize noise and restore specific images. The essence of predicted image generation is that the model can accurately and gradually remove part of the noise from the complete noise to obtain a predicted image; then, in order to improve the speed of model application, the image features corresponding to the current state of the robotic arm are input into the model during actual use to obtain multiple predicted noise diffusion features obtained by diffusion, and then the predicted noise diffusion features can be fused based on the additionally deployed lightweight task graph to obtain the corresponding action features more quickly, and the action features can be decoded to obtain specific control signals for controlling the robotic arm.

[0033] refer to Figure 2 , Figure 2 A flowchart of extracting image trajectory features in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; in some embodiments, extracting multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from a preset robotic arm action video in sequence includes but is not limited to the following steps: Step S210, performing video frame sampling and division on the preset robot arm action video according to the video frame time sequence, to obtain a plurality of sample historical image frames and a sample target image frame located after the plurality of sample historical image frames; Step S220: extracting sample historical image trajectory features from sample historical image frames and extracting sample target image trajectory features from sample target image frames through a preset video feature extractor.

[0034] Among them, according to the time sequence of the video frames, the pre-set robot arm action video is sampled and divided, and multiple sample historical image frames and sample target image frames are extracted therefrom. It can be understood that continuous video image sampling is performed to ensure that the model can learn the change rules of image sequences from the past to the future, laying the foundation for subsequent feature extraction and model training, and then the preset video feature extractor can be used to extract sample historical image trajectory features from the sample historical image frames, and extract sample target image trajectory features from the sample target image frames.

[0035] In some embodiments, the sample historical image frame is an image frame extracted from the video and represents the movement of the robotic arm in a certain period of time in the past, and the sample target image frame is an image frame immediately following the sample historical image frame and represents the movement of the robotic arm in a certain period of time in the future.

[0036] In some embodiments, the video feature extractor may be a deep learning model such as a convolutional neural network (CNN), which is used to extract useful features from images; the sample historical image trajectory features may be features extracted from sample historical image frames, which reflect the movements of the robotic arm in past time periods; the sample target image trajectory features may be features extracted from sample target image frames, which reflect the movements of the robotic arm in future time periods.

[0037] In some embodiments, video frame sampling and division are performed on a preset robot arm action video according to the video frame timing to obtain multiple sample historical image frames and sample target image frames located after the multiple sample historical image frames. This can include uniformly sampling 4 frames of the robot video according to the sampling ratio, wherein the first two frames are used as historical frames and the last two frames are used as future frames, and then encoding these frames into representations through a video extractor.

[0038] refer to Figure 3 , Figure 3 A flow chart of obtaining a sample trajectory noise diffusion feature in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; in some embodiments, noise diffusion is performed on sample historical image trajectory features and sample target image trajectory features to obtain a sample trajectory noise diffusion feature, including but not limited to the following steps: Step S310, combining the sample history image trajectory feature and the sample target image trajectory feature into a sample image trajectory feature; Step S320, adding preset noise data to the sample image trajectory feature in sequence according to the time step order to obtain the sample trajectory noise diffusion feature.

[0039] Among them, the sample historical image trajectory features and the sample target image trajectory features are merged to form a complete sample image trajectory feature, which can facilitate subsequent noise diffusion processing, so that the model can process historical and future image features at the same time under the same framework; in the order of time steps, preset noise data is gradually added to the sample image trajectory features to generate sample trajectory noise diffusion features. It can be understood that noise diffusion is a method to enhance the generalization ability of the model. By gradually adding noise, the training model learns to recover clear image features from the noise, thereby improving the robustness and generalization ability of the model, so that it can maintain good performance when facing new data.

[0040] In some embodiments, the sample image trajectory feature is a comprehensive feature representation formed by combining the sample historical image trajectory feature and the sample target image trajectory feature; noise diffusion is the process of gradually adding preset noise data to the sample image trajectory feature in the order of time steps to generate the sample trajectory noise diffusion feature; the sample trajectory noise diffusion feature is the sample image trajectory feature after noise diffusion processing, which is used to perform positive noise diffusion training on the model.

[0041] refer to Figure 4 , Figure 4 In the robot arm control method based on the video diffusion model provided in one embodiment of the present application, a flow chart of obtaining a target video diffusion model is provided; in some embodiments, based on the sample historical image trajectory features and the sample trajectory noise diffusion features, the noise diffusion features associated with the sample target image are used as the training target, and the pre-trained preset multi-frame generation model is trained to obtain a target video diffusion model for outputting the sample target image, including but not limited to the following steps: Step S410, performing zero-filling of the sample history image trajectory feature in the dimension corresponding to the sample target image, so that the filled sample history image trajectory feature is connected with the sample trajectory noise diffusion feature along the channel dimension to obtain the noise diffusion condition feature; Step S420 trains the pre-trained preset multi-frame generation model based on the noise diffusion condition feature and the noise diffusion feature associated with the sample target image as a training target to obtain a target video diffusion model for outputting the sample target image.

[0042] Among them, the sample historical image trajectory features are filled with zero values ​​of the corresponding dimensions of the sample target image, so that the filled sample historical image trajectory features and the sample trajectory noise diffusion features can be connected in the channel dimension to form noise diffusion conditional features. It can be understood that by filling zero values ​​in certain dimensions of the feature tensor, the sizes of different feature tensors are ensured to match, so as to facilitate subsequent feature connection operations to create conditional inputs and help the model better understand the relationship between historical and future images; further, based on the noise diffusion conditional features, the pre-trained multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target to obtain a target video diffusion model for outputting the sample target image. It can be understood that by using a deep learning framework, the pre-trained multi-frame generation model can be fine-tuned and the parameters of the model can be optimized so that it can predict future image features based on historical image features, and then accurately predict future image features from historical image features, thereby improving the accuracy of robot arm motion prediction.

[0043] In some embodiments, the sample historical image trajectory features are filled with zero values ​​of the corresponding dimensions of the sample target image to ensure that the sizes of different feature tensors match, and the sample historical image trajectory features after zero value filling are connected with the sample trajectory noise diffusion features to form a conditional input feature for model training; during model training, the present application can be based on the noise diffusion conditional features, with the noise diffusion features associated with the sample target image as the training target, to train the pre-trained multi-frame generation model to obtain a target video diffusion model for outputting the sample target image; the noise diffusion conditional features are the input features for model training, which include historical image features and noise diffusion features; the noise diffusion features associated with the sample target image are the features used as training targets, representing the noise diffusion version of future image frames; the pre-trained multi-frame generation model is a model that has been trained using a large amount of data and can generate continuous multi-frame images; the target video diffusion model is a model that is obtained by fine-tuning the pre-trained multi-frame generation model and can predict future image features based on historical image features.

[0044] refer to Figure 5 , Figure 5 In the robot arm control method based on the video diffusion model provided in one embodiment of the present application, another flow chart of obtaining the target video diffusion model; in some embodiments, based on the noise diffusion condition feature, the noise diffusion feature associated with the sample target image is used as the training target, and the pre-trained preset multi-frame generation model is trained to obtain the target video diffusion model for outputting the sample target image, including but not limited to the following steps: Step S510, obtaining language instruction features for instructing the pre-trained preset multi-frame generation model to predict the sample target image; Step S520, obtaining the diffusion time step corresponding to the noise diffusion condition feature; Step S530, input the noise diffusion condition features, language instruction features and diffusion time steps into the multi-frame generation model, take the noise diffusion features associated with the sample target image as the training target, train the preset multi-frame generation model, and obtain the target video diffusion model for outputting the sample target image.

[0045] In some embodiments, in the process of obtaining language instruction features used to instruct a pre-trained preset multi-frame generation model to predict a sample target image, features can be extracted from text instructions through natural language processing technology. These features are used to guide the model to generate specific image content. For example, the instruction "move the robotic arm to position A" can be converted into a feature in the form of a vector for use by the model.

[0046] In some embodiments, the diffusion time step represents a time point or step in the noise diffusion process. In the noise diffusion process, as the time step increases, the noise gradually increases, and the model needs to learn to recover clear image features from the noise at different time steps. By inputting the noise diffusion condition features, language instruction features, and diffusion time steps into the multi-frame generation model, the pre-trained multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target, and a target video diffusion model for outputting the sample target image is obtained.

[0047] Among them, the language instruction features are features extracted from text instructions, which are used to guide the model to generate specific image content. The noise diffusion features associated with the sample target image are used as the features of the training target, representing the noise diffusion version of the future image frame. Then, by fine-tuning the pre-trained multi-frame generation model, a model is obtained that can predict future image features based on historical image features and language instructions.

[0048] In some embodiments, a preset multi-frame generation model is trained with the noise diffusion feature associated with the sample target image as a training target, including: obtaining a predicted noise diffusion feature obtained by the preset multi-frame generation model based on the diffusion of the sample historical image; performing model loss calculation based on the predicted noise diffusion feature and the noise diffusion feature associated with the sample target image, so as to update the model parameters according to the obtained model loss. It can be understood that the sample historical image feature is processed through a noise diffusion process to generate a predicted noise diffusion feature with noise, and the predicted noise diffusion feature is the predicted feature generated by the multi-frame generation model according to the noise diffusion condition feature, which represents the model's prediction of future images; and then the model loss calculation can be performed based on the predicted noise diffusion feature and the noise diffusion feature associated with the sample target image. The noise diffusion feature associated with the sample target image is used as the feature of the training target, which represents the noise diffusion version of the future image frame, and can be used to calculate the difference between the predicted noise diffusion feature and the noise diffusion feature associated with the sample target image. The optimization algorithm such as gradient descent is used to adjust the parameters of the model according to the calculated loss function value to reduce the loss and improve the prediction accuracy of the model.

[0049] In some embodiments, the updating process of the above-mentioned model parameters can be understood as the process of forward diffusion training the model's cognitive ability of noise. After the model parameters are updated, the video diffusion model can also be used for the reverse denoising process, that is, the video diffusion model can be used to perform diffusion noise removal iteration based on the diffusion time step from the preset Gaussian noise to obtain a sample target image corresponding to the language instruction feature.

[0050] refer to Figure 6 , Figure 6In the robot arm control method based on the video diffusion model provided in one embodiment of the present application, a schematic diagram of a target video diffusion model is obtained; corresponding to Figure 6 It can be understood that, in the first stage corresponding to steps S110 to S130, the present application can connect the video representation processed by the diffusion process with the historical video representation along the channel dimension to form . Together with language tags and diffusion steps Input Open-Sora together for diffusion generation training.

[0051] In some embodiments, corresponding to the above steps S110 to S130, in order to enable the model to have knowledge of world dynamics, the present application describes this knowledge acquisition stage as the ability to generate future image trajectories. This ability requires the model to learn external dynamic priors and use Bayes' theorem, which can be further expressed as: .

[0052] in, is the likelihood of observation under a given state, given the known observation image When , the probability of the corresponding state S is , S* represents the optimal state sequence, is the prior probability of the state sequence. This formula is essential in many applications, including hidden Markov models and other state estimation techniques. As more observations are added, the likelihood More concentrated around the true state sequence. This is because more data points can better distinguish different state sequences.

[0053] According to the above formula, in order to more accurately predict future state transitions, the goal of the corresponding stages of step S110 to step S130 of this application is to predict future video frames based on historical video frames and language instructions. To achieve this goal, this application uses the multi-frame generation model Open-Sora of the Video Diffusion Transformer (VDT): This model performs well in generating diverse and physically realistic continuous video frames that are aligned with language instructions. For the sake of simplicity, this application uses VDT to represent Open-Sora in the following content.

[0054] In some embodiments, corresponding to step S110 to step S130, specifically, in order to prepare training data, such as Figure 5 As shown, the present application can use the pre-trained video representation extractor to extract continuous robot images in the trajectory. Encoding as Representation ,in and Respectively represent Sample historical images and future images corresponding to the sample target image In the forward diffusion process, the original data is gradually transformed into noise data. According to the diffusion step Added trajectory representation The diffusion model is optimized to predict the added noise from a given conditional cue. As a conditional reminder, this application will Characterized by zero value Fill to The same shape and along the channel dimension Connecting to form conditional visual representations . Language Instructions is encoded by a text tokenizer

[11] to obtain a language representation. Mathematically, the above denoising diffusion model is optimized as: .

[0055] Based on the above It can be seen from the expression that steps S110 to S130 are used to fill the historical image with zero values ​​to instruct the model to generate images based on the historical images and text instructions, and then compare the generated future image with noise with the image with noise diffusion, with the goal of minimizing the difference between the predicted noise and the actual noise, and optimizing the model parameters so that the model learns how to generate noise. Then, the model can reversely recover the original data from the future image with noise to output relatively accurate image predictions, thereby completing model training.

[0056] In some embodiments, the preset robotic arm action video uses a third-person perspective camera to predict the representation. It is understandable that most robotic datasets only contain third-person perspective data. Training with a fixed third-person perspective can reduce the impact of perspective changes and help the model focus on predicting the transformation of the robotic arm itself.

[0057] refer to Figure 7 , Figure 7 A flowchart of obtaining predicted noise diffusion features in a robotic arm control method based on a video diffusion model provided in an embodiment of the present application; in some embodiments, the current image trajectory features of the robotic arm are input into a target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, including but not limited to the following steps: Step S710, the current image trajectory feature of the robot arm is filled with zero values ​​based on the corresponding dimension of the sample target image, and the filled current image trajectory feature is connected with the Gaussian noise of the same dimension along the channel dimension to obtain the current visual feature; Step S720: input the current visual features and the corresponding language instruction features into the video diffusion model, and set the diffusion time step of the video diffusion model to a preset maximum time step to obtain a plurality of predicted noise diffusion features obtained by diffusion.

[0058] In some embodiments, the current image trajectory feature can be understood as a feature extracted from the current image of the robot arm, indicating the current image state, and filling certain dimensions of the feature tensor with zero values ​​to ensure that the sizes of different feature tensors match; Gaussian noise can be understood as randomly generated noise data used to simulate the noise in the diffusion process, and the current visual feature can be understood as the input feature formed by connecting the current image trajectory feature after zero value filling with Gaussian noise.

[0059] In some embodiments, the language instruction feature is a feature extracted from text instructions, which is used to guide the model to generate specific image content. The diffusion time step represents the time point or step in the noise diffusion process. Setting it to the maximum time step means that the model will make predictions in the most complex noise environment. The predicted noise diffusion feature represents the predicted feature generated by the model based on the current visual features and language instruction features, which represents the model's prediction of future images.

[0060] In some embodiments, a multi-layer cascaded action adapter included in the target video diffusion model performs feature fusion based on multiple predicted noise diffusion features to obtain multiple predicted action features, including: reading the predicted noise diffusion features of each layer of the video diffusion model through a multi-layer cascaded action adapter; performing feature fusion on multiple predicted noise diffusion features based on the multi-head cross-attention query features preset in the action adapter to obtain multiple predicted action features, wherein the feature fusion process can be understood as performing feature fusion based on multiple predicted noise diffusion features through the multi-layer cascaded action adapter included in the target video diffusion model to obtain multiple predicted action features, and the multi-layer cascaded action adapter is a lightweight module for fusing the features of each layer of the video diffusion model to generate the final action representation. It can be understood that the predicted features generated by the video diffusion model represent the model's prediction of future images, so the multi-head cross-attention query features can be preset in the action adapter for feature fusion of multiple predicted noise diffusion features. In addition, a front network with a gating mechanism can also be configured in the action adapter for feature data screening.

[0061] It can be understood that the above steps together constitute a complete process of generating predicted action features from the current image trajectory features and language command features of the robot arm. Through the above steps, the model can generate current visual features suitable for the input video diffusion model through zero-value filling and Gaussian noise addition, input the current visual features and language command features into the video diffusion model to generate predicted noise diffusion features, and then through multi-layer cascaded action adapters and multi-head cross-attention mechanisms, perform feature fusion on multiple predicted noise diffusion features to generate predicted action features.

[0062] refer to Figure 8 , Figure 8 In a robotic arm control method based on a video diffusion model provided in an embodiment of the present application, a flowchart of obtaining multiple motion control signals is provided; in some embodiments, motion decoding is performed on multiple predicted motion features according to control parameters of the robotic arm to obtain multiple motion control signals, including but not limited to the following steps: Step S810, obtaining control parameters of the robot arm and adapter parameters of the motion adapter; Step S820, configuring decoder parameters of the motion decoder according to the control parameters; constructing a decoding loss function according to the model parameters, the decoder parameters and the adapter parameters; Step S830, updating decoder parameters based on the decoding loss corresponding to the decoding loss function, so that the action decoder performs action decoding on the multiple predicted action features to obtain multiple action control signals.

[0063] In some embodiments, the control parameters of the robotic arm include various physical parameters and control parameters of the robotic arm, such as joint angles, speeds, accelerations, etc., which are used to configure the motion decoder; the adapter parameters of the motion adapter are parameters learned by the multi-layer cascaded motion adapters during the training process, which are used for feature fusion and generation of predicted motion features.

[0064] In some embodiments, decoder parameters of the action decoder are configured according to the control parameters; a decoding loss function is constructed according to the model parameters, the decoder parameters, and the adapter parameters. The decoder parameters are the initial parameters of the action decoder, which are used to convert the predicted action features into specific action control signals, and then the decoder parameters can be updated based on the decoding loss corresponding to the decoding loss function, so that the action decoder can perform action decoding on multiple predicted action features to obtain multiple action control signals. The action control signal is a specific action control signal generated by the decoder, which is used to control the robot arm to perform a specific action.

[0065] In some embodiments, the action decoder can perform action estimation based on a diffusion model, which is equivalent to performing feature fusion denoising on the noise after diffusion of historical images in each layer of the model in the action adapter of implicit inverse dynamics to obtain the action representation, and then denoising the action representation through the action decoder to generate an executable action.

[0066] refer to Fig. 9 , Fig. 9 In the robotic arm control method based on the video diffusion model provided in one embodiment of the present application, a schematic diagram of multiple motion control signals is obtained; corresponding to the above steps S140 to S150, according to the inverse dynamics representation:

[0067] From observation to state, in many practical applications, direct access to the state space is not feasible, a series of image observations are provided instead. Representing the observation space, such as images and sensor data, from a series of observations Infer the underlying state sequence from , actions can be accurately predicted in the state with an inverse dynamics model; it is understandable that a direct way to combine the inverse dynamics model with the video diffusion transform architecture (VDT) learned in the first stage is to separately build an inverse dynamics model from image observations to actions. During deployment, the model can predict actions from observations generated in the dynamic perception vision stage. However, in this way, actions can only be predicted after the VDT undergoes a time-consuming iterative denoising diffusion process, which is not ideal for high-frequency robot control. In addition, the accuracy of the action depends heavily on the accuracy of the predicted observations. Since not all pixels are important for predicting actions, this approach may introduce unnecessary bias and time cost. In addition, learning a separate inverse dynamics model from scratch cannot take advantage of the pre-trained parameters of the VDT.

[0068] Therefore, in order to solve the above problems, this application proposes to directly adapt VDT to an inverse dynamics model so that the dynamic knowledge and implicit states learned by VDT can be seamlessly utilized to promote action prediction, that is, the lightweight action adapter introduced in this application, which can effectively convert VDT into an implicit inverse dynamics model. The output of the action adapter is then decoded into a series of actions through a diffusion-based action head.

[0069] In some embodiments, corresponding to Fig. 9 It is understandable that in the second stage corresponding to steps S140 to S150, the present application can introduce a learnable layer-by-layer adapter into the Open-Sora trained in the first stage. . And use a learnable action query representation Integrate the intermediate estimates of the model to obtain the final action representation Then Input Action Diffusion Decoder Conduct motion prediction training.

[0070] In some embodiments, it is understood that in order to convert the VDT into an inverse dynamics model, the present application introduces a layer-by-layer action adapter after each layer of the VDT. Each action adapter contains a multi-head cross attention and a feed-forward network with a gating mechanism. The present application uses Learnable action query representation As a query for the cross-attention block, the knowledge of each layer of the VDT is fused to generate Final action representation Since this application only reuses VDT parameters without using its observation generation function, this application uses a fixed diffusion step to disable the iterative denoising process, which will is transformed into pure Gaussian noise. Formally, the action representation is parameterized by a layer-by-layer cross-attention adapter as Fusion: .

[0071] in, represents the VDT parameters combined with the layer-by-layer cross-attention adapter parameters The flexibility of the layer-by-layer action adapter allows domain-specific knowledge to be integrated into the action representation. For example, if the downstream robot includes wrist camera images, this application uses an image encoder followed by a perceptual resampler to encode them into wrist markers, which are concatenated with the output of each layer of the VDT as additional keys and values ​​for the layer-by-layer adapter.

[0072] In some embodiments, it can be understood that the action decoder is used to perform action decoding based on the diffusion model, and the fused action representation It is then converted into low-level control signals, the action decoder Responsible for decoding the action representation into executable action signals, such as determining the 7 degrees of freedom (DoF) of the end effector posture and the gripper state. Similar to the dynamic perception vision stage, the goals of action prediction are: .

[0073] in , represents the diffusion step, Represents the forward diffusion process. It should be noted that, compared with VDT, the diffusion-based action decoder is relatively small, so its computational cost and time consumption are almost negligible; it can be understood that the VDT model gradually adds noise through the forward diffusion process, and then gradually removes the noise through the reverse generation process to generate future image representations. Although this process can generate high-quality future images, it requires multiple iterations of denoising, and the computational cost is high; if the VDT model completes the prediction and obtains the image results in the dynamic perception visual stage, and then performs action prediction based on these image results, each prediction will require a complete denoising process, which is time-consuming. Therefore, the diffusion-based action decoder can effectively avoid the time-consuming iterative denoising process in the dynamic perception visual stage, thereby reducing deviations and time costs.

[0074] In some embodiments, during deployment, the present application forward adapts the VDT once to obtain the action representation , without iterative generation process, according to the above formula , then the action decoder performs an iterative denoising process to Convert to executable actions.

[0075] In some embodiments, the present application uses the RLBench dataset as training data for imitation learning, and selects the CoppeliaSim simulation environment as a test platform. RLBench contains 15 tasks, each with 2-60 variants. For example, "stacking 2 red blocks" and "stacking 4 purple blocks" in the stacking block task are considered to be two different variants. During testing, the model needs to handle new object poses, randomly sampled targets, and object colors, shapes, sizes, and categories with different semantic instantiations. Each multi-task agent independently evaluates all tasks, with an evaluation score of 0 indicating failure and 100 indicating complete success. This application reports the average success rate of each task in 25 evaluation scenarios, for agents trained with 10 examples (denoted as RLBench-10) and agents trained with 100 examples (denoted as RLBench-100); in addition, when performing offline evaluation on data, this application also reports offline indicators including the average and mean square error (MSE) of the position xyz estimation accuracy and the Euler angle estimation accuracy (Avgxyzang). These indicators measure the accuracy of the robot's predicted 3D position and orientation angle compared to the true value; the following conclusions can be drawn: The method VidMan of this application performs well in both simulation environments and offline evaluations, especially in cases where data is limited. Through a two-stage training strategy, the method of this application can more effectively utilize training data and improve task success rate and accuracy.

[0076] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application provides a robot arm control device based on a video diffusion model, including: a model training module, used to extract multiple sample historical image trajectory features and multiple sample target image trajectory features located after the multiple sample historical image trajectory features from a preset robot arm action video; perform noise diffusion on the sample historical image trajectory features and the sample target image trajectory features to obtain sample trajectory noise diffusion features; based on the sample historical image trajectory features and the sample trajectory noise diffusion features, take the noise diffusion features associated with the sample target image as the training target, train a pre-trained preset multi-frame generation model to obtain a target video diffusion model for outputting the sample target image; an action control module, used to input the current image trajectory features of the robot arm into the target video diffusion model to obtain multiple predicted noise diffusion features obtained by diffusion, and perform feature fusion based on the multiple predicted noise diffusion features through the multi-layer cascaded action adapter contained in the target video diffusion model to obtain multiple predicted action features; and perform feature fusion on the multiple predicted action features according to the control parameters of the robot arm.

[0077] Some embodiments of the present application provide a computer device, Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application, referring to Fig.10 The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the robot arm control method based on the video diffusion model of any of the above embodiments is implemented, for example, the above described Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Steps S510 to S530 of the method, Figure 7 Steps S710 to S720 of the method, Figure 8 Method steps S810 to S830.

[0078] The computer device 1000 of the embodiment of the present application includes one or more processors 1010 and a memory 1020. Fig.10 In the figure, a processor 1010 and a memory 1020 are taken as an example.

[0079] The processor 1010 and the memory 1020 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0080] The memory 1020, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1020 may optionally include a memory 1020 remotely disposed relative to the processor 1010, and these remote memories may be connected to the computer device 1000 via a network. At the same time, examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0081] In some embodiments, when the processor executes the computer program, the robot arm control method based on the video diffusion model of any of the above embodiments is executed at preset intervals.

[0082] Those skilled in the art will understand that Fig.10 The device structure shown in the figure does not constitute a limitation on the computer device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] exist Fig.10 In the computer device 1000 shown, the processor 1010 can be used to call the robotic arm control method based on the video diffusion model stored in the memory 1020, so as to implement the robotic arm control method based on the video diffusion model.

[0084] Based on the hardware structure of the above-mentioned computer device 1000, various embodiments of the robotic arm control device based on the video diffusion model of the present application are proposed. At the same time, the non-transient software program and instructions required to implement the robotic arm control method based on the video diffusion model of the above-mentioned embodiment are stored in the memory. When executed by the processor, the robotic arm control method based on the video diffusion model of the above-mentioned embodiment is executed.

[0085] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned robot arm control method based on the video diffusion model, which can enable the above-mentioned one or more processors to execute the robot arm control method based on the video diffusion model of any of the above-mentioned embodiments, for example, to execute the above-mentioned Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5Steps S510 to S530 of the method, Figure 7 Steps S710 to S720 of the method, Figure 8 Method steps S810 to S830.

[0086] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the robot arm control method based on the video diffusion model to implement any one of the above embodiments, for example, executing the above described Figure 1 Steps S110 to S150 of the method, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Steps S510 to S530 of the method, Figure 7 Steps S710 to S720 of the method, Figure 8 The method steps S810 to S830 in the above device embodiment are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD to ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A robotic arm control method based on a video diffusion model, characterized in that: include: Extracting a plurality of sample historical image trajectory features and a plurality of sample target image trajectory features after the plurality of sample historical image trajectory features from a preset robot arm motion video; Performing noise diffusion on the sample history image trajectory feature and the sample target image trajectory feature to obtain a sample trajectory noise diffusion feature; Based on the sample historical image trajectory features and the sample trajectory noise diffusion features, the noise diffusion features associated with the sample target image are used as training targets, and a pre-trained preset multi-frame generation model is trained to obtain a target video diffusion model for outputting the sample target image; Inputting the current image trajectory feature of the robot arm into the target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, and performing feature fusion based on the plurality of predicted noise diffusion features through a multi-layer cascaded action adapter included in the target video diffusion model to obtain a plurality of predicted action features; The plurality of predicted motion features are subjected to motion decoding according to the control parameters of the robotic arm to obtain a plurality of motion control signals, so as to control the robotic arm based on the plurality of motion control signals.

2. The robotic arm control method based on the video diffusion model according to claim 1, characterized in that: The extracting of a plurality of sample historical image trajectory features and a plurality of sample target image trajectory features after the plurality of sample historical image trajectory features from the preset robot arm motion video comprises: Performing video frame sampling and division on a preset robot arm motion video according to a video frame time sequence to obtain a plurality of sample historical image frames and a sample target image frame located after the plurality of sample historical image frames; The sample historical image trajectory features are extracted from the sample historical image frames by a preset video feature extractor, and the sample target image trajectory features are extracted from the sample target image frames.

3. The robotic arm control method based on the video diffusion model according to claim 1, characterized in that: The performing noise diffusion on the sample historical image trajectory feature and the sample target image trajectory feature to obtain the sample trajectory noise diffusion feature includes: Combining the sample history image trajectory feature and the sample target image trajectory feature into a sample image trajectory feature; Preset noise data is added to the sample image trajectory feature in sequence according to the time step order to obtain the sample trajectory noise diffusion feature.

4. The robotic arm control method based on video diffusion model according to claim 1, characterized in that: The method of training a pre-trained preset multi-frame generation model based on the sample historical image trajectory feature and the sample trajectory noise diffusion feature and taking the noise diffusion feature associated with the sample target image as a training target to obtain a target video diffusion model for outputting the sample target image includes: Filling the sample history image trajectory feature with zero values ​​of the dimension corresponding to the sample target image, so that the filled sample history image trajectory feature is connected with the sample trajectory noise diffusion feature along the channel dimension to obtain the noise diffusion condition feature; Based on the noise diffusion condition characteristics, the noise diffusion characteristics associated with the sample target image are used as training targets, and the pre-trained preset multi-frame generation model is trained to obtain a target video diffusion model for outputting the sample target image.

5. The robotic arm control method based on the video diffusion model according to claim 4, characterized in that: The method of training a pre-trained preset multi-frame generation model based on the noise diffusion condition feature and taking the noise diffusion feature associated with the sample target image as a training target to obtain a target video diffusion model for outputting the sample target image includes: Obtaining language instruction features for instructing a pre-trained preset multi-frame generation model to predict a sample target image; Obtaining a diffusion time step corresponding to the noise diffusion condition feature; The noise diffusion condition features, the language instruction features and the diffusion time step are input into the multi-frame generation model, and the noise diffusion features associated with the sample target image are used as training targets to train the preset multi-frame generation model to obtain a target video diffusion model for outputting the sample target image.

6. The robotic arm control method based on the video diffusion model according to claim 5, characterized in that: The method of training the preset multi-frame generation model by taking the noise diffusion feature associated with the sample target image as the training target includes: Obtaining predicted noise diffusion features obtained by the preset multi-frame generation model based on sample historical image diffusion; A model loss calculation is performed according to the predicted noise diffusion feature and the noise diffusion feature associated with the sample target image, so as to update the model parameters according to the obtained model loss.

7. The robotic arm control method based on video diffusion model according to claim 1, characterized in that: The step of inputting the current image trajectory feature of the robot arm into the target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion includes: Filling the current image trajectory feature of the robot arm with zero values ​​based on the corresponding dimension of the sample target image, and connecting the filled current image trajectory feature with Gaussian noise of the same dimension along the channel dimension to obtain the current visual feature; The current visual features and the corresponding language instruction features are input into the video diffusion model, and the diffusion time step of the video diffusion model is set to a preset maximum time step to obtain a plurality of predicted noise diffusion features obtained by diffusion.

8. The robotic arm control method based on video diffusion model according to claim 1, characterized in that: The multi-layer cascaded action adapter included in the target video diffusion model performs feature fusion based on the multiple predicted noise diffusion features to obtain multiple predicted action features, including: Reading the predicted noise diffusion features of each layer of the video diffusion model through a multi-layer cascaded action adapter; Based on the multi-head cross-attention query features preset in the action adapter, feature fusion is performed on the multiple predicted noise diffusion features to obtain multiple predicted action features.

9. The robotic arm control method based on video diffusion model according to claim 8, characterized in that: The step of performing action decoding on the plurality of predicted action features according to the control parameters of the robot arm to obtain a plurality of action control signals includes: Acquiring control parameters of the robot arm and adapter parameters of the motion adapter; configuring decoder parameters of the action decoder according to the control parameters; constructing a decoding loss function according to the model parameters, the decoder parameters and the adapter parameters; The decoder parameters are updated based on the decoding loss corresponding to the decoding loss function, so that the action decoder performs action decoding on the plurality of predicted action features to obtain a plurality of action control signals.

10. A robotic arm control device based on a video diffusion model, characterized in that: include: A model training module, used to extract a plurality of sample historical image trajectory features and a plurality of sample target image trajectory features located after the plurality of sample historical image trajectory features from a preset robot arm action video; Performing noise diffusion on the sample history image trajectory feature and the sample target image trajectory feature to obtain a sample trajectory noise diffusion feature; Based on the sample historical image trajectory features and the sample trajectory noise diffusion features, the pre-trained preset multi-frame generation model is trained with the noise diffusion features associated with the sample target image as the training target to obtain a target video diffusion model for outputting the sample target image; The motion control module is used to input the current image trajectory features of the robot arm into the target video diffusion model to obtain a plurality of predicted noise diffusion features obtained by diffusion, and perform feature fusion based on the plurality of predicted noise diffusion features through the multi-layer cascaded motion adapter included in the target video diffusion model to obtain a plurality of predicted motion features; perform motion decoding on the plurality of predicted motion features according to the control parameters of the robot arm to obtain a plurality of motion control signals, so as to control the robot arm based on the plurality of motion control signals.

11. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the robot arm control method based on the video diffusion model as described in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the robotic arm control method based on the video diffusion model as described in any one of claims 1 to 9.

Images