Robot control method and robot

Abstract

The invention provides a robot control method and a robot, and the method comprises the steps: achieving the end-to-end closed loop of attitude estimation, action generation and robot control through a unified conditional diffusion model. The method comprises the following steps: firstly, combining a robot working image and a noise-containing attitude vector, and performing denoising processing to obtain accurate attitude information; generating a sparse video frame based on the working image and the task instruction, and splitting the sparse video frame into continuous frame pairs; and finally, denoising the noisy action sequence through a conditional diffusion model to obtain a dense target action sequence, and controlling the robot to execute by taking the attitude information as a reference. According to the method, an additional auxiliary mode or an independent module is not needed, the system architecture is simplified, the generated action sequence is continuous, smooth and physically executable, the precision and efficiency of robot control are greatly improved, the stability of task execution is guaranteed, and the method is suitable for scenes such as human-computer interaction and object control.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a robot control method and a robot. Background Technology

[0002] Robot pose estimation and inverse dynamics modeling are core components of robot control, widely applied in human-robot interaction, multi-robot collaboration, and object manipulation. The former requires accurate extraction of 3D key points or joint angles from visual observations, while the latter requires converting task-related videos into a continuous sequence of executable actions. With the successful application of diffusion models in robotics, a unified framework is urgently needed to address the high-precision requirements of single-image pose estimation and the high-efficiency requirements of mapping sparse videos to actions.

[0003] Existing robot pose estimation techniques fall into two categories. The first relies on markers in images or real-time joint feedback, combined with camera intrinsic parameters to calculate pose. The second predicts depth maps / 2D keypoints and then converts them into 3D poses to achieve joint-information-free estimation. Regarding inverse dynamics modeling, existing methods are mostly independent modules, based on video frames generated from world models, outputting actions through iterative inference, with some employing diffusion strategies to optimize action generation.

[0004] However, the multi-stage process of attitude estimation limits accuracy and speed, and relies on auxiliary modes or prior information; existing inverse dynamics models are mostly offline-online hybrid models, generating sparse and discontinuous motion sequences with high latency. The lack of a unified framework for both types of tasks results in high system complexity, making it difficult to meet the stability requirements of real-time robot control and long-term task execution. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a robot control method and a robot, thereby solving the problems of inconsistent task frameworks for existing posture estimation and inverse dynamics modeling, high system complexity of robots, and difficulty in meeting the stability requirements of real-time control and long-term task execution of robots.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a robot control method, the method comprising: Based on the robot's working image and the acquired robot posture vector, the robot's posture information is obtained. The robot's working image includes: the robot's working part and the environmental state. Sparse video frames are generated based on the robot's working images and task instructions, and the sparse video frames are split into multiple consecutive video frame pairs; Each video frame pair is input into a pre-trained conditional diffusion model, and the conditional diffusion model performs denoising processing on the noisy action sequence based on each video frame pair to obtain the robot's target action sequence. The robot is controlled to perform motion control based on the target motion sequence and the posture information.

[0007] Optionally, obtaining the robot's posture information based on the robot's working image and the acquired robot posture vector includes: Based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model, the robot's pose vector is denoised to obtain the robot's pose information.

[0008] Optionally, the step of denoising the robot's pose vector based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model to obtain the robot's pose information includes: The robot's working image is encoded based on the conditional diffusion model to obtain image features; The first diffusion time step is encoded based on the conditional diffusion model to obtain the first temporal feature; The pose vector is denoised using the conditional diffusion model based on the image features and the first temporal features to obtain the robot's pose information.

[0009] Optionally, the process by which the conditional diffusion model denoises the pose vector based on the image features and the first temporal features to obtain the robot's pose information includes: The image features and the first temporal features are concatenated to obtain the first conditional features; The first conditional feature is subjected to feature linear modulation to obtain a first modulation feature, which includes: a scale modulation feature vector and an offset modulation feature vector. The attitude vector is encoded to obtain attitude features; The attitude feature is linearly modulated based on the first modulation feature to obtain the processed feature; The processed features and the pose features are combined to obtain the pose information.

[0010] Optionally, splitting the sparse video frame into multiple consecutive pairs of video frames includes: Starting from the first video frame of the sparse video frames, the sparse video frames are traversed using a sliding window, and the video frames within the sliding window are combined into a video frame pair. The sliding window is moved according to a preset step size and a preset moving direction to obtain multiple video frame pairs.

[0011] Optionally, the conditional diffusion model performs denoising processing on the noisy action sequence based on each of the video frames to obtain the robot's target action sequence, including: Based on the conditional diffusion model, the video frames in the video frame pair are encoded to obtain video frame features; The second diffusion time step is randomly sampled and encoded based on the conditional diffusion model to obtain the second temporal feature; Based on the video frame features and the second temporal features, the noisy action sequence is denoised by the conditional diffusion model to obtain the target action sequence.

[0012] Optionally, the conditional diffusion model performs denoising processing on the noisy action sequence based on the video frame features and the second temporal features to obtain the target action sequence, including: The video frame features and the second temporal features are concatenated to obtain the second conditional features; The second conditional feature is subjected to feature linear modulation to obtain the second modulation feature, which includes: scale modulation feature vector and bias modulation feature vector. The noisy action sequence is encoded to obtain action features; The action feature is linearly modulated according to the second modulation feature to obtain the modulated feature; The modulated features and the action features are combined to obtain an initial action sequence; The initial action sequence is deduplicated to obtain the target action sequence.

[0013] Optionally, the step of deduplicating the initial action sequence to obtain the target action sequence includes: Identify at least one overlapping action group in the initial action sequence; Traverse each overlapping action group, determine the average value of each overlapping action group, and use the average value as the overlapping action corresponding to the overlapping action group to obtain the target action sequence.

[0014] Optionally, the step of controlling the robot's motion based on the target action sequence and the posture information includes: Based on the posture information, the robot is controlled to execute each action in the target action sequence in the order of the target action sequence, so as to achieve the target posture of the target action sequence.

[0015] Secondly, this application provides a robot for performing the steps of a robot control method as described in any one of the first aspects.

[0016] The beneficial effects of this application are as follows: An end-to-end closed loop of attitude estimation, motion generation, and robot control is achieved through a unified conditional diffusion model. High-precision attitude information is obtained by combining robot working images with diffusion denoising technology, thus providing a reliable benchmark for control. Sparse video frames are generated by the world model and split into frame pairs, enabling accurate capture of key state transitions in the task. Noise-free motion sequences are denoised and overlapping motions are optimized using the conditional diffusion model, generating dense and continuous target motion sequences. Furthermore, offline control is achieved based on attitude information and target motion sequences, eliminating the need for real-time environmental feedback, balancing control accuracy and efficiency, significantly improving the robot's task success rate, reducing system complexity and control latency, and making it suitable for various robot application scenarios such as human-robot interaction and object manipulation.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this application is shown; Figure 2 A flowchart of a robot control method provided in an embodiment of this application is shown; Figure 3 This document illustrates a flowchart of a robot pose estimation method provided in an embodiment of this application. Figure 4 This document illustrates a flowchart of yet another method for robot pose estimation provided in an embodiment of this application. Figure 5 This document illustrates a flowchart of a method for splitting video frames into multiple pairs, as provided in an embodiment of this application. Figure 6 A flowchart illustrating an embodiment of this application for generating an action sequence is shown; Figure 7 A flowchart illustrating yet another method for generating an action sequence, as provided in an embodiment of this application, is shown. Figure 8 This document illustrates a flowchart of an action deduplication process provided in an embodiment of this application. Figure 9 This paper shows a schematic diagram of the structure of a robot control device provided in an embodiment of this application; Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Current technologies for attitude estimation suffer from limitations in accuracy and speed due to their multi-stage processes, and rely on auxiliary modalities or prior information. Furthermore, inverse dynamics models generate sparse, discontinuous motion sequences with high latency, often operating in a hybrid offline-online mode. In addition, the lack of a unified framework for both types of tasks leads to high system complexity, making it difficult to meet the stability requirements of real-time robot control and long-term task execution.

[0023] Based on this, this application proposes a conditional diffusion model that can simultaneously achieve the two core tasks of robot pose estimation and motion sequence generation. In the following embodiments of this application, robot pose estimation and robot motion sequence generation are described as a complete task. However, it should be understood that robot pose estimation and robot motion sequence generation can be two independent tasks. For example, robot pose estimation can be performed solely based on the conditional diffusion model, or robot motion sequence generation can be performed solely based on the conditional diffusion model. Alternatively, they can be two sequential sub-tasks during robot task execution. The specific usage of the model is not limited here. For example, first, robot pose estimation can be performed based on the conditional diffusion model, and then robot motion sequence generation can be performed based on the conditional diffusion model. During the execution of the robot motion sequence, the robot pose can be estimated simultaneously using the conditional diffusion model, enabling the robot to perform the corresponding actions according to the generated robot motion sequence.

[0024] First, the working scenario of the robot in this application will be described. (Refer to...) Figure 1In this embodiment, the camera used to acquire images can be a standalone camera or a camera integrated into the robot. The robot includes a robotic arm for executing specific action commands. In the case of a camera integrated into the robot, the robot also includes a camera for acquiring images.

[0025] Figure 1 This is a schematic diagram of a robot performing a task. The robot's robotic arm can perform task-related actions such as grasping and moving target objects in the task scene. The camera can capture working images of the robotic arm performing the task in the task scene. In addition, the robot also includes various sensors and data acquisition devices. The processor in the robot can estimate the robot's posture and generate the robot's action sequence based on the robot control method of this application, according to the acquired data and working images.

[0026] In another possible implementation, a standalone camera can also communicate with the robot and record images of the robot working in the work scene from a third-party perspective. These images are then sent to the robot's processor for processing. This application does not limit the specific implementation method.

[0027] Next, combine Figure 2 The robot control method of this application will be described, such as... Figure 2 As shown, the method includes: S201. Obtain the robot's posture information based on the robot's working image and the acquired robot posture vector.

[0028] The robot working image includes both the robot's working details and the environmental state. The robot working image is an RGB image used to reflect the robot's working state in the task scenario. (Refer to...) Figure 1 The robot's working parts can be the robot's robotic arm joints and end effectors, while the environmental state includes the position of the target object and the layout of the task scene. For example, when a robot performs the task of picking up a bottle, the robot's working image is an image that includes the bottle, the robotic arm, and the basket in which the bottle is placed.

[0029] A robot pose vector is a quantitative representation of the robot's true state in three-dimensional space, including the coordinates of 3D key points and joint angles. It should be noted that due to sensor errors and environmental interference, the robot pose vector contains noise interference. Therefore, it is necessary to perform denoising processing on the robot pose vector using images of the robot at work to obtain clean robot pose information.

[0030] Among them, the robot posture information is the robot posture data obtained after denoising processing by the conditional diffusion model. It is the result of restoration and optimization of the noisy posture vector, which is highly close to the real robot posture vector and provides a reference starting point for motion control.

[0031] S202. Generate sparse video frames based on the robot's working images and task instructions, and split the sparse video frames into multiple consecutive video frame pairs.

[0032] The task instructions can be natural language text or natural language speech describing the robot's target task, used to guide the world model to generate task-related videos. For example, a language instruction can be given to the robot: "Pick up the salad dressing and put it in the basket." The robot can then recognize the speech and convert it into text to obtain the natural language text instruction.

[0033] After receiving task instructions, the robot can generate a sequence of visual frames based on the robot's working image and the task instructions using a pre-deployed world model. This results in sparse video frames, which only contain key task steps (such as the initial state, grasping state, and placement state). These frames are discontinuous and cannot be directly used for robot control. Therefore, the sparse video frames can be processed in conjunction with the robot's working image to obtain a continuous and dense sequence of actions that the robot can execute.

[0034] Sparse video frames consist of multiple video frames, each indicating a posture the robot should achieve. By parsing the robot's posture within the video frames, the robot's action sequence can be obtained, and the robot can be controlled to execute the corresponding actions according to the action sequence. The sparse video frames are split into multiple consecutive video frame pairs according to the order of the individual frames. Each video frame pair consists of two consecutive video frames: the first video frame of the current frame is the same as the second video frame of the previous frame, and the second video frame of the current frame is the same as the first video frame of the next frame.

[0035] S203. Input each video frame pair into the pre-trained conditional diffusion model. The conditional diffusion model performs denoising on the noisy action sequence based on each video frame pair to obtain the robot's target action sequence.

[0036] Among them, the conditional diffusion model can be a unified model that is built on the denoising diffusion probability model, adopts the Conditional U-Net architecture and FiLM modulation mechanism, and can simultaneously realize pose denoising and action sequence generation. After offline training, it has end-to-end mapping capability.

[0037] Noisy motion sequences refer to randomly generated high-dimensional noise vectors with dimensions consistent with the target motion sequence (determined by the robot's degrees of freedom and motion block length). They simulate the state of real motion sequences affected by noise and serve as the initial input for diffusion denoising. The target motion sequence is a dense and continuous set of motion instructions obtained after denoising using a conditional diffusion model and optimization by averaging overlapping motions. It can directly drive the robot to complete tasks and possesses both physical executability and temporal coherence.

[0038] Noisy action sequences are denoised step-by-step using a diffusion model, gradually reconstructing continuous actions that conform to visual observation (video frame pairs). The video frame pairs provide inter-frame state constraints, guiding the model to generate dense actions that match state transitions, thus solving the mapping problem from sparse frames to actions.

[0039] In one possible implementation, the conditional diffusion model adapted to each video frame can be used as input. The model encodes two frames separately and concatenates them into video frame features. A second temporal feature is obtained by randomly sampling and encoding the second diffusion time step. The concatenated features are then modulated to generate a second modulation feature. Next, noisy action sequences are encoded to obtain action features. After linear modulation and residual combination, initial action blocks are generated through multi-step diffusion denoising. Finally, overlapping actions of adjacent action blocks are identified, and the average value is calculated to eliminate bias. The concatenated blocks then yield a complete and smooth target action sequence.

[0040] S204. Perform motion control on the robot based on the target action sequence and posture information.

[0041] Among them, posture information serves as the baseline starting point for action execution, and the target action sequence can provide step-by-step posture change instructions. The combination of the two can ensure that the robot smoothly transitions from the initial state to the target state, achieving offline precise control without real-time feedback.

[0042] In one possible implementation, the posture information can be calibrated with the initial state of the robot's own sensors to ensure consistency of the execution starting point. The target action sequence can be converted into a format supported by the robot controller, and the kinematic constraints can be verified. Then, based on the posture information, action commands are issued one by one in sequence. The posture transition is achieved by incremental superposition. After the execution is completed, the consistency between the final posture and the task target is verified to complete the offline closed-loop control.

[0043] This application's embodiments achieve an end-to-end closed loop of attitude estimation, motion generation, and robot control through a unified conditional diffusion model. High-precision attitude information is obtained by combining robot working images with diffusion denoising technology, thus providing a reliable benchmark for control. Sparse video frames are generated by a world model and split into frame pairs, enabling accurate capture of key task state transitions. Noise-free motion sequences are denoised and overlapping motions are optimized using the conditional diffusion model, generating dense and continuous target motion sequences. Furthermore, offline control is achieved based on attitude information and target motion sequences, eliminating the need for real-time environmental feedback. This balances control accuracy and efficiency, significantly improving robot task success rate, reducing system complexity and control latency, and making it suitable for various robot application scenarios such as human-robot interaction and object manipulation.

[0044] The following is a further explanation of the process of obtaining robot posture information based on robot working images and acquired robot posture vectors. Step S201 includes: Based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model, the robot's pose vector is denoised to obtain the robot's pose information.

[0045] It should be understood that noisy pose vectors cannot be directly used as the reference for robot control. The denoising capability of the conditional diffusion model is required. The visual features of the robot's working image are used as the core input and the first diffusion time step is used as the temporal feature. Gaussian noise is gradually eliminated through the back diffusion process to restore the pure pose information that highly matches the robot's true pose.

[0046] In one possible implementation, the robot's working image can be first input into a pre-trained conditional diffusion model. The model's built-in encoder extracts features from the image to obtain image features reflecting the robot's working local state and the environment. Then, the first diffusion time step is randomly sampled from the model's preset noise schedule and embedded and encoded by the model's encoder to generate temporal features. Finally, the image features and temporal features are used as conditional inputs to the denoising network of the conditional diffusion model. The model follows the reverse denoising process of the denoising diffusion probability model, predicts and eliminates the Gaussian noise injected into the noisy posture vector step by step, and finally outputs the denoised and clean robot posture information.

[0047] In another possible implementation, the robot's working image can be preprocessed with lightweight scalability and then input into the feature extraction branch of the conditional diffusion model to obtain image features that fuse the environment and robot state. The first diffusion time step is then encoded at multiple scales to generate a first temporal feature containing information from different time steps. The image features and the multi-scale first temporal feature are then concatenated and linearly modulated to obtain the modulated conditional feature. This feature, along with the noisy pose vector, is then input into the model's denoising branch. The feature representation is enhanced through the residual module, and the robot's pose information is gradually denoised.

[0048] The following further explains the process of denoising the robot's pose vector based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model to obtain the robot's pose information. Figure 3 As shown, the above steps include: S301. Encode the robot's working image based on the conditional diffusion model to obtain image features.

[0049] Image features are visual feature vectors extracted from robot working images after being encoded by a conditional diffusion model. They contain key visual information about the robot's working area and the state of its environment.

[0050] Optionally, the conditional diffusion model includes an image encoder, which can input the robot's working image into the image encoder for convolution, pooling, and other operations to extract visual features layer by layer from shallow to deep layers, and finally output fixed-dimensional image features that fully preserve key information about the robot's working local area and environmental state. The image encoder can be, for example, a ResNet encoder.

[0051] In another possible implementation, the robot's working image can be preprocessed by normalization and deblurring before being input into the lightweight coding branch of the conditional diffusion model. Image features can be extracted by replacing traditional convolution with depthwise separable convolution, which reduces the amount of computation while ensuring the effectiveness of the features. The output image features have the same feature dimension as the core implementation method and can be directly used for subsequent feature fusion.

[0052] S302. Encode the first diffusion time step based on the conditional diffusion model to obtain the first time-series feature.

[0053] Optionally, the conditional diffusion model includes a time-step encoder, which can randomly sample the first diffusion time step from the noise schedule preset by the conditional diffusion model, input the discrete value into the time-step encoder of the model, and encode it through sine and cosine position embedding to generate a first temporal feature that matches the dimension of the image features. This feature can be directly spliced ​​and fused with the image features.

[0054] In another possible implementation, the first diffusion time step of random sampling can be processed by multi-level mapping and then input into the time step encoder of the model. Encoding is performed by stacking fully connected layers to generate the first temporal feature containing multi-scale information of the time step, thereby improving the model's adaptability to different noise stages.

[0055] S303. The robot's posture vector is denoised using the conditional diffusion model based on image features and the first temporal features to obtain the robot's posture information.

[0056] In one possible implementation, image features can be directly concatenated with first temporal features to form conditional features. These conditional features are then input into the conditional encoder of the conditional diffusion model. The conditional encoder linearly modulates the posture features obtained by encoding noisy posture information based on the conditional features. The residual module is then added for combined processing. Finally, the model predicts and eliminates noise through a reverse denoising process, and outputs clean robot posture information after iterating step by step.

[0057] In another possible implementation, the image features and the first temporal features can be fused together to obtain the first conditional features. The conditional encoder generates a modulation vector and modulates the pose features. Attention mechanism can be introduced to allow the model to focus on key joints or key point information in the pose features. Then, diffusion denoising iteration is performed to further improve the accuracy of pose denoising and finally obtain the robot's pose information.

[0058] In this embodiment, image features and first temporal features are extracted separately through model encoding, and the two features are combined to complete pose denoising, preserving both visual and temporal constraint information. This makes the diffusion denoising process more targeted and effectively improves the accuracy of pose information extraction. Simultaneously, the step-by-step encoding method makes feature extraction more efficient, and the denoising logic that integrates the two features conforms to the working principle of the diffusion model, ensuring the robustness of pose denoising. Furthermore, the above process is implemented through a unified conditional diffusion model, requiring no additional modules, reducing system complexity, and providing a high-precision and reliable pose reference for subsequent robot motion control.

[0059] The following is a further explanation of the process by which the conditional diffusion model denoises the robot's pose vector based on image features and the first temporal feature to obtain the robot's pose information. Figure 4 As shown, the above step S303 includes: S401. Concatenate the image features and the first temporal features to obtain the first conditional features.

[0060] Optionally, the encoded fixed-dimensional image features can be directly concatenated with the first temporal features of the same dimension along the channel dimension. That is, the vectors of the two features are concatenated column by column to form a higher-dimensional feature vector, generating the first conditional feature, which retains all the core information of both visual and temporal aspects.

[0061] As another possible implementation, the image features and the first temporal features can be normalized first to eliminate differences in feature numerical scales, and then the features can be concatenated in the spatial dimension. After concatenation, the dimensions are compressed through a convolutional layer to obtain the first conditional features with dimension adaptation, thereby improving the efficiency of feature fusion.

[0062] S402. Perform feature linear modulation on the first condition feature to obtain the first modulation feature, which includes: scale modulation feature vector and bias modulation feature vector.

[0063] Optionally, the first conditional feature can be input into a dedicated conditional encoder of the conditional diffusion model. The encoder is composed of stacked fully connected layers. Through the parameters learned by the model training, it directly outputs a scale modulation feature vector and a bias modulation feature vector that match the pose feature dimension. The two together constitute the first modulation feature.

[0064] S403. Encode the attitude vector to obtain attitude features.

[0065] Among them, the noisy attitude vector is a high-dimensional original vector that cannot be directly operated with the modulation features. It needs to be transformed into abstract attitude features through model encoding, and the numerical information of the attitude is transformed into a feature representation that the model can process.

[0066] Alternatively, the noisy pose vector can be input into the pose encoder of the conditional diffusion model. The encoder extracts features and maps dimensions of the high-dimensional pose vector through convolutional and pooling layers, and outputs pose features of fixed dimensions. These features fully preserve the core state of the noisy pose information.

[0067] S404. Linearly modulate the attitude feature according to the first modulation feature to obtain the processed feature.

[0068] The scale and bias vector of the first modulation feature are generated based on visual and temporal conditions. By scaling and offsetting the pose features element by element, the pose features can be incorporated into the conditional constraint information, so that the subsequent denoising process can be carried out based on the robot's actual working state and diffusion time step, avoiding pose estimation deviation caused by unconstrained denoising.

[0069] In one possible implementation, the scale modulation feature vector in the first modulation feature can be multiplied element-wise with the attitude feature, and then added element-wise with the bias modulation feature vector to obtain the processed feature with fusion condition constraints.

[0070] S405. Combine the processed features and pose features to obtain pose information.

[0071] In one possible implementation, a residual combination approach can be used. First, the original pose features are input into a convolutional layer for dimension adaptation to make them consistent with the dimensions of the processed features. Then, the dimension-adapted pose features and the processed features are added element-wise. The combined features are then input into the Conditional U-Net denoising branch of the Conditional Diffusion model. After time-step diffusion denoising, clean robot pose information is output.

[0072] The processed features are modulation features that incorporate conditional constraints. The original pose features retain the basic representation of noisy pose information. The combination of the two can achieve complementary enhancement of features, avoid the loss of original pose information during modulation, and improve the model's feature learning ability through residual connections, so that the model can more accurately restore the pure pose information from the modulated features.

[0073] In this embodiment, through step-by-step operations of splicing, linear modulation, encoding, and combination processing, visual and temporal conditions are precisely integrated into the pose denoising process, effectively improving the extraction accuracy of pose information. Linear modulation and residual combination processing preserve core pose information, enhance the model's feature representation capabilities, and prevent gradient vanishing.

[0074] The following is a further explanation of the above method of splitting sparse video frames into multiple consecutive pairs of video frames, such as... Figure 5 As shown, the above step S202 includes: S501. Starting from the first video frame of the sparse video frames, a sliding window is used to traverse the sparse video frames and combine the video frames in the sliding window into a video frame pair.

[0075] A sliding window is a fixed-length data capture window used to traverse sparse video frames. The length of the sliding window can be 2 to accommodate two consecutive video frames.

[0076] Sparse video frames contain only discrete frames of key steps in the task and lack continuous state transition information. It is necessary to start from the first frame and traverse it sequentially, using a fixed-length sliding window to extract two consecutive frames to form a frame pair, capturing the state differences between adjacent key steps, and providing basic input for the generation of subsequent dense actions.

[0077] In one possible implementation, the temporal order of sparse video frames can be determined. Starting from the first video frame, a sliding window with a length of 2 is initialized. The first two consecutive sparse video frames covered by the window are directly combined to form the first video frame pair. The window maintains a fixed length and continues to traverse subsequent sparse video frames along the temporal direction, ensuring that each window contains two consecutive video frames.

[0078] Optionally, starting from the first video frame of the sparse video frames, the timing of the sparse video frames can be checked first to remove invalid frames with out-of-order frames, and then a sliding window with a window length of 2 can be used to traverse the valid timing sequence.

[0079] S502. Move the sliding window according to the preset step size and preset movement direction to obtain multiple video frame pairs.

[0080] The preset step size refers to the number of frames the sliding window moves through in each sparse video frame sequence. The preset step size can be, for example, 1. The preset movement direction is a fixed direction in which the sliding window traverses the sparse video frames, which is a forward temporal direction from the first frame to the last frame of the sparse video frames, ensuring that the frame pairs conform to the time logic of task execution.

[0081] In one possible implementation, assuming the preset sliding window step size is 1 and the moving direction is the forward temporal direction from the first frame to the last frame, after generating the first video frame pair, the sliding window is moved 1 frame in the preset direction by the preset step size, so that the window covers the second and third frames and combines them into a new video frame pair. This moving and combining operation is repeated until the end of the sliding window covers the last frame of the sparse video frames, resulting in multiple consecutive video frame pairs that cover all sparse frames.

[0082] This application embodiment achieves efficient conversion from discrete keyframes to continuous video frame pairs by splitting sparse video frames according to fixed rules using a sliding window. It accurately captures the state differences between adjacent key steps, providing effective input for subsequent action generation. By presetting the step size and movement direction, it ensures the continuity and temporal rationality of frame pairs and allows adjacent frame pairs to overlap, providing a foundation for deduplication and optimization of subsequent action sequences. Furthermore, the splitting process is simple and efficient, without complex feature processing, reducing computational costs and enabling rapid adaptation to sparse video frames of different lengths, thus improving the versatility of the solution.

[0083] The following is a further explanation of how the conditional diffusion model above performs denoising on noisy action sequences based on each video frame to obtain the robot's target action sequence. Figure 6 As shown, the above step S203 includes: S601. Encode the video frames in the video frame pair based on the conditional diffusion model to obtain video frame features.

[0084] Video frame features refer to the visual feature vectors extracted from a single frame in a video frame pair after being encoded by a conditional diffusion model. They contain key information about the robot's working area, environmental state, and object position within a single video frame. The fusion of video frame features from two frames can reflect the state changes between video frames.

[0085] Optionally, the preceding and following frames in a video frame pair can be input into a pre-trained conditional diffusion model. The model's built-in ResNet encoder performs convolution and pooling operations on each frame sequentially, extracting visual features from shallow to deep layers, and outputting fixed-dimensional video frame features for the preceding and following frames, thus fully preserving the state information of each frame.

[0086] S602. Randomly sample the second diffusion time step and encode the second diffusion time step based on the conditional diffusion model to obtain the second time series feature.

[0087] The second diffusion time step is a diffusion time step used for denoising the action sequence. It is randomly sampled from the pre-set noise schedule of the conditional diffusion model and sampled independently from the first diffusion time step of attitude estimation. It is used to guide the stepwise denoising process of the noisy action sequence.

[0088] The second temporal feature is a temporal representation obtained after the second diffusion time step is encoded by the conditional diffusion model. It reflects the noise scheduling stage of action sequence denoising, provides temporal guidance for noisy action sequence denoising, and is fused with video frame features to form conditional constraints for action denoising.

[0089] Optionally, a second diffusion time step can be randomly sampled from the noise schedule preset by the conditional diffusion model. This discrete value is then input into the step encoder of the model and encoded using a sine-cosine position embedding method to generate a second temporal feature that matches the feature dimension of the video frame. This feature can then be directly fused with the video frame feature.

[0090] S603. Based on the video frame features and the second temporal features, the noisy action sequence is denoised using the conditional diffusion model to obtain the target action sequence.

[0091] Optionally, the features of the previous and next frames of the video frame pair can be concatenated and fused, and then concatenated with the second temporal features to form the conditional features for motion denoising. These features are then input into the conditional encoder of the conditional diffusion model. The scale and bias modulation vectors are generated through linear modulation. The motion features obtained by encoding the noisy motion sequence are linearly modulated. After adding the residual module for combined processing, the model predicts and eliminates noise step by step, and finally obtains the target motion sequence.

[0092] This application embodiment obtains video frame features and second temporal features through encoding. The fusion of the two features provides accurate visual and temporal constraints for action denoising, effectively improving the matching degree between action sequences and inter-frame state transitions. Relying on the denoising capability of the conditional diffusion model, a continuous and smooth action sequence is generated, solving the problem of discontinuity in the mapping from sparse frames to actions.

[0093] The following is a further explanation of how the conditional diffusion model above denoises noisy action sequences based on video frame features and second temporal features to obtain the target action sequence. Figure 7 As shown, the above step S603 includes: S701. Concatenate the video frame features and the second temporal features to obtain the second conditional features.

[0094] S702. Perform characteristic linear modulation on the second conditional feature to obtain the second modulation feature, which includes: scale modulation feature vector and bias modulation feature vector.

[0095] S703. Encode the noisy action sequence to obtain action features.

[0096] S704. Linearly modulate the action feature according to the second modulation feature to obtain the modulated feature.

[0097] S705. Combine the modulated features and action features to obtain the initial action sequence.

[0098] S706. Perform deduplication on the initial action sequence to obtain the target action sequence.

[0099] The second conditional feature is a fusion feature formed by concatenating the video frame features and the second temporal feature. It contains inter-frame visual state constraints and temporal guidance information for action denoising, and serves as the input for feature linear modulation in action sequence denoising. The second modulation feature is the feature obtained after linear modulation of the second conditional feature. It includes scale modulation feature vector and bias modulation feature vector, and is specifically used to linearly modulate the encoded features of noisy action sequences.

[0100] Action features are high-dimensional abstract features extracted after encoding noisy action sequences, including the basic temporal and state information of the noisy action sequences. Modulated features are features obtained by linearly modulating the action features with a second modulation feature, integrating visual, temporal constraints, and action-related information.

[0101] The initial action sequence is obtained by combining the modulated features and action features and then performing diffusion denoising. It does not perform overlap deduplication optimization and there is overlap deviation between adjacent action blocks.

[0102] Optionally, the fixed-dimensional video frame features obtained from encoding can be directly concatenated with the second temporal features of the same dimension in the channel dimension to generate a higher-dimensional feature vector, generating a second conditional feature. This second conditional feature is then input into the dedicated conditional encoder of the conditional diffusion model. Through a stacked fully connected layer encoder, based on the parameters learned during training, it outputs scale-modulated feature vectors and bias-modulated feature vectors that match the dimension of the action features. The noisy action sequence is then input into the action encoder of the conditional diffusion model. Convolutional and pooling layers are used to extract features and map dimensions of the high-dimensional action vector, outputting fixed-dimensional action features. These features are calculated using the formula: Modulated Feature = Scale-Modulated Feature Vector ⊙ Action Feature + Bias-Modulated Feature Vector (⊙ is element-wise multiplication). First, the scale vector is multiplied element-wise with the action feature, then element-wise added with the bias vector to obtain the fused conditional modulated feature. A residual combination method is then used to input the original action feature into a convolutional layer for dimension adaptation, making it consistent with the dimension of the modulated feature. The two are then added element-wise, and the combined feature is input into the Conditional Diffusion Model. The U-Net denoising branch, after being denoised step-by-step by DDPM, outputs an unoptimized initial action sequence.

[0103] The following is a further explanation of the deduplication process performed on the initial action sequence to obtain the target action sequence, as described above. Figure 8 As shown, the above step S706 includes: S801. Determine at least one overlapping action group in the initial action sequence.

[0104] In this context, an overlapping action group refers to a set of one or more actions in the initial action sequence that overlap with each other, generated from action blocks of adjacent frame pairs. Since sparse video frame splitting uses a sliding window with a step size of 1, adjacent action blocks must have at least one overlapping action. For example, the last action of the action block of frame pair (I1, I2) and the first action of the action block of frame pair (I2, I3) form an overlapping action group.

[0105] In one possible implementation, the splicing order of each action block in the initial action sequence can be determined according to the splitting rules of sparse video frames. According to the rule that the last action of the previous action block is the first action of the next action block, all adjacent action blocks are traversed, and the corresponding overlapping actions are marked and extracted one by one. Each marked overlapping action independently constitutes an overlapping action group.

[0106] S802. Traverse each overlapping action group, determine the average value of each overlapping action group, and use the average value as the overlapping action corresponding to the overlapping action group to obtain the target action sequence.

[0107] Optionally, all the determined overlapping action groups can be traversed in sequence, and the arithmetic mean of all action values ​​in each overlapping action group can be calculated to obtain the average value of the group. The calculated average value can then be used to replace all the original action values ​​of the corresponding overlapping action group in the initial action sequence. After optimizing all overlapping actions, the action blocks can be reassembled in time sequence to obtain a target action sequence that is unbiased, smooth and continuous.

[0108] In this embodiment, by identifying overlapping action groups and calculating the average value to replace the original overlapping actions, the deviation in action values ​​caused by different frame pairs in the prediction is effectively eliminated, allowing adjacent action blocks to be smoothly connected, ensuring the continuity and smoothness of the target action sequence, and avoiding sudden changes in action during robot execution.

[0109] After obtaining the target action sequence, the robot can be controlled to move based on the target action sequence and posture information. Step S204 above includes: Based on the attitude information, the robot is controlled to execute each action in the target action sequence in the order of the target action sequence in order to achieve the target attitude of the target action sequence.

[0110] Optionally, the posture information can be calibrated with the initial state of the robot's own sensors to determine the execution reference. Then, the target action sequence can be converted into a format supported by the robot controller, and action commands can be issued one by one at fixed time intervals in chronological order. Each action command is executed based on the reference posture increment, and the end posture of the previous action is used as the starting point for the execution of the next action. The actions are gradually superimposed until all actions are completed and the target posture is achieved.

[0111] It should be noted that during the execution of the target action sequence, the robot can also continuously acquire robot working images and perform robot pose estimation based on the robot working images. The pose of the target action sequence can be calibrated based on the estimated pose, thereby improving the accuracy of robot control.

[0112] Based on the same inventive concept, this application also provides a robot control device corresponding to the robot control method. Since the principle of the device in this application is similar to that of the robot control method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0113] Figure 9 A schematic diagram of a robot control device provided in an embodiment of this application is shown. The device includes an acquisition module 901, a generation module 902, a noise reduction module 903, and a control module 904.

[0114] The acquisition module 901 is used to acquire the robot's posture information based on the robot's working image and the acquired robot posture vector. The robot's working image includes: the robot's working part and the environmental state. The generation module 902 is used to generate sparse video frames based on robot working images and task instructions, and to split the sparse video frames into multiple consecutive video frame pairs. The denoising module 903 is used to input each video frame pair into a pre-trained conditional diffusion model. The conditional diffusion model performs denoising processing on the noisy action sequence based on each video frame pair to obtain the robot's target action sequence. The control module 904 is used to control the robot's movements based on the target action sequence and posture information.

[0115] Optionally, module 901 is specifically used for: Based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model, the robot's pose vector is denoised to obtain the robot's pose information.

[0116] Optionally, the noise reduction module 903 is specifically used for: Image features are obtained by encoding robot working images based on a conditional diffusion model; The first diffusion time step is encoded based on the conditional diffusion model to obtain the first temporal feature; The pose vector is denoised using a conditional diffusion model based on image features and first temporal features to obtain the robot's pose information.

[0117] Optionally, the noise reduction module 903 is specifically used for: The image features and the first temporal features are concatenated to obtain the first conditional features; The first conditional feature is subjected to feature linear modulation to obtain the first modulation feature, which includes: scale modulation feature vector and bias modulation feature vector; The attitude vector is encoded to obtain attitude features; The attitude features are linearly modulated based on the first modulation feature to obtain the processed features; The processed features and pose features are combined to obtain pose information.

[0118] Optionally, the generation module 902 is specifically used for: Starting from the first video frame of the sparse video frames, a sliding window is used to traverse the sparse video frames and combine the video frames in the sliding window into a video frame pair. The sliding window is moved according to the preset step size and preset movement direction to obtain multiple video frame pairs.

[0119] Optionally, the noise reduction module 903 is specifically used for: Video frames in a video frame pair are encoded based on a conditional diffusion model to obtain video frame features; The second diffusion time step is randomly sampled and encoded based on the conditional diffusion model to obtain the second temporal feature; Based on video frame features and second temporal features, the noisy action sequence is denoised using a conditional diffusion model to obtain the target action sequence.

[0120] Optionally, the noise reduction module 903 is specifically used for: The video frame features and the second temporal features are concatenated to obtain the second conditional features; The second conditional feature is subjected to feature linear modulation to obtain the second modulation feature, which includes: scale modulation feature vector and bias modulation feature vector. Noisy action sequences are encoded to obtain action features; The motion feature is linearly modulated based on the second modulation feature to obtain the modulated feature; The modulated features and action features are combined to obtain the initial action sequence; The initial action sequence is deduplicated to obtain the target action sequence.

[0121] Optionally, the noise reduction module 903 is specifically used for: Identify at least one overlapping action group in the initial action sequence; Iterate through each overlapping action group, determine the average value of each overlapping action group, and use the average value as the overlapping action corresponding to the overlapping action group to obtain the target action sequence.

[0122] Optionally, the control module 904 is specifically used for: Based on the attitude information, the robot is controlled to execute each action in the target action sequence in the order of the target action sequence in order to achieve the target attitude of the target action sequence.

[0123] This application achieves an end-to-end closed loop of attitude estimation, motion generation, and robot control through a unified conditional diffusion model. High-precision attitude information is obtained by combining robot working images with diffusion denoising technology, thus providing a reliable benchmark for control. Sparse video frames are generated by a world model and split into frame pairs, enabling accurate capture of key task state transitions. Noise-free motion sequences are denoised and overlapping motions are optimized using the conditional diffusion model, generating dense and continuous target motion sequences. Furthermore, offline control is achieved based on attitude information and target motion sequences, eliminating the need for real-time environmental feedback. This balances control accuracy and efficiency, significantly improving robot task success rate, reducing system complexity and control latency, and making it suitable for various robot applications such as human-robot interaction and object manipulation.

[0124] Figure 10 This illustration shows a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device can communicate with a robot or be deployed inside the robot to execute steps in a robot control method. The electronic device includes a processor 1001, a storage medium 1002, and a bus 1003. The storage medium 1002 stores machine-readable instructions executable by the processor 1001. When the electronic device runs a robot control method as described in the embodiment, the processor 1001 communicates with the storage medium 1002 via the bus 1003. The processor 1001 executes the machine-readable instructions, and the preamble of the method item of the processor 1001 executes the steps in the above-mentioned robot control method.

[0125] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor, and the processor performs the steps in the above-described robot control method.

[0126] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0130] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0132] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A robot control method, characterized in that, include: Based on the robot's working image and the acquired robot posture vector, the robot's posture information is obtained. The robot's working image includes: the robot's working part and the environmental state. Sparse video frames are generated based on the robot's working images and task instructions, and the sparse video frames are split into multiple consecutive video frame pairs; Each video frame pair is input into a pre-trained conditional diffusion model, and the conditional diffusion model performs denoising processing on the noisy action sequence based on each video frame pair to obtain the robot's target action sequence. The robot is controlled to perform motion control based on the target motion sequence and the posture information.

2. The method according to claim 1, characterized in that, The step of obtaining the robot's posture information based on the robot's working image and the acquired robot posture vector includes: Based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model, the robot's pose vector is denoised to obtain the robot's pose information.

3. The method according to claim 2, characterized in that, The step of denoising the robot's pose vector based on the robot's working image, the first diffusion time step, and the pre-trained conditional diffusion model to obtain the robot's pose information includes: The robot's working image is encoded based on the conditional diffusion model to obtain image features; The first diffusion time step is encoded based on the conditional diffusion model to obtain the first temporal feature; The pose vector is denoised using the conditional diffusion model based on the image features and the first temporal features to obtain the robot's pose information.

4. The method according to claim 3, characterized in that, The process by which the conditional diffusion model denoises the pose vector based on the image features and the first temporal features to obtain the robot's pose information includes: The image features and the first temporal features are concatenated to obtain the first conditional features; The first conditional feature is subjected to feature linear modulation to obtain a first modulation feature, which includes: a scale modulation feature vector and an offset modulation feature vector. The attitude vector is encoded to obtain attitude features; The attitude feature is linearly modulated based on the first modulation feature to obtain the processed feature; The processed features and the pose features are combined to obtain the pose information.

5. The method according to claim 1, characterized in that, The step of splitting the sparse video frame into multiple consecutive video frame pairs includes: Starting from the first video frame of the sparse video frames, the sparse video frames are traversed using a sliding window, and the video frames within the sliding window are combined into a video frame pair. The sliding window is moved according to a preset step size and a preset moving direction to obtain multiple video frame pairs.

6. The method according to claim 1, characterized in that, The conditional diffusion model denoises the noisy action sequence based on each video frame pair to obtain the robot's target action sequence, including: Based on the conditional diffusion model, the video frames in the video frame pair are encoded to obtain video frame features; The second diffusion time step is randomly sampled and encoded based on the conditional diffusion model to obtain the second temporal feature; Based on the video frame features and the second temporal features, the noisy action sequence is denoised by the conditional diffusion model to obtain the target action sequence.

7. The method according to claim 6, characterized in that, The conditional diffusion model denoises the noisy action sequence based on the video frame features and the second temporal features to obtain the target action sequence, including: The video frame features and the second temporal features are concatenated to obtain the second conditional features; The second conditional feature is subjected to feature linear modulation to obtain the second modulation feature, which includes: scale modulation feature vector and bias modulation feature vector. The noisy action sequence is encoded to obtain action features; The action feature is linearly modulated according to the second modulation feature to obtain the modulated feature; The modulated features and the action features are combined to obtain an initial action sequence; The initial action sequence is deduplicated to obtain the target action sequence.

8. The method according to claim 7, characterized in that, The process of deduplicating the initial action sequence to obtain the target action sequence includes: Identify at least one overlapping action group in the initial action sequence; Traverse each overlapping action group, determine the average value of each overlapping action group, and use the average value as the overlapping action corresponding to the overlapping action group to obtain the target action sequence.

9. The method according to claim 1, characterized in that, The step of controlling the robot's motion based on the target motion sequence and the posture information includes: Based on the posture information, the robot is controlled to execute each action in the target action sequence in the order of the target action sequence, so as to achieve the target posture of the target action sequence.

10. A robot, characterized in that, include: The robot is used to perform the steps of a robot control method as described in any one of claims 1 to 9.

Images