Robot cross-view action control method and system based on spatiotemporal view synthesis, and robot

Abstract

This invention relates to the field of visual control technology and discloses a method, system, and robot for cross-viewpoint motion control based on spatiotemporal perspective synthesis. It addresses the technical problem in existing technologies where changes in viewpoint cause motion deviations in visually controlled robots. The method includes: acquiring a real-time visual image from the current viewpoint during a control cycle; estimating the corresponding depth image and relative pose of the viewpoint; constructing a first point cloud based on the depth image and camera parameters, and reprojecting the first point cloud onto a preset viewpoint based on the relative pose of the viewpoint to obtain a second point cloud, and generating a coarsely rendered image from the preset viewpoint; generating historical temporal conditions based on the synthesized image from the previous control cycle; calling a visual diffusion model to generate a synthesized image from the preset viewpoint during the current control cycle; extracting observation features generated by the visual diffusion model during the diffusion process, and inputting the observation features and robot state features into a motion policy network for motion prediction to obtain a motion control sequence.

Description

Technical Field

[0001] This invention relates to the field of visual control technology, and in particular to a method, system, computer storage medium, computer program product, and robot for cross-view motion control based on spatiotemporal perspective synthesis. Background Technology

[0002] In recent years, the scalability and generalization capabilities of robot manipulation technologies implemented through visual control have attracted attention, such as robots based on imitation learning visual servoing strategies and robots implemented using Vision Language Action (VLA) technology. These robots can achieve visual servoing functions through the correspondence between visual representations and motion control during pre-training. However, these robots often use a fixed camera perspective during training, which changes during testing or in actual operation. Changes in perspective disrupt the spatial correspondence between visual representations and motion control, and in severe cases, can lead to significant strategy failure.

[0003] In existing technologies, some solutions employ 3D structure reconstruction based on multi-view information, while others use prediction of future frames. However, these solutions suffer from low inference efficiency and insufficient success rate in practical tasks, particularly in closed-loop control tasks. Therefore, there is an urgent need for a robot motion control method with cross-view generalization capabilities, capable of accurately planning and controlling robot movements even when changes in viewpoint cause shifts in visual feature distribution. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem in the prior art where changes in the viewpoint cause a shift in the distribution of visual features, leading to motion deviations in visually controlled robots and making motion control processes prone to failure.

[0005] Technical effect: When the visual feature distribution shifts due to changes in viewing angle, it can infer the visual information of the current scene under a identifiable preset viewing angle based on real-time visual image information, and perform motion planning and control of the robot, thereby improving the viewing angle generalization ability and enabling cross-view visual image processing.

[0006] The robot described in this embodiment refers to an intelligent machine capable of semi-autonomous or fully autonomous operation, and its shape is not restricted.

[0007] The first aspect of this invention provides a method for cross-view robot motion control based on spatiotemporal perspective synthesis, comprising:

[0008] In the current control cycle, acquire real-time visual images from the current perspective;

[0009] Based on the real-time visual image, the geometric estimation model is invoked to estimate the depth image and view-relative pose at the current viewpoint.

[0010] The first point cloud is constructed based on the depth image and camera parameters at the current viewpoint, and the first point cloud is reprojected onto the preset viewpoint based on the relative pose of the viewpoint to obtain the second point cloud.

[0011] A coarsely rendered image from a preset viewpoint is generated based on the second point cloud;

[0012] Encode the synthetic image generated in the previous control cycle from the preset viewpoint to generate historical time sequence conditions;

[0013] The coarse-rendered image is encoded, and a visual diffusion model is invoked to generate a synthetic image with a preset viewpoint under the current control cycle based on the encoded coarse-rendered image and diffusion conditions. The diffusion conditions include the historical time sequence conditions.

[0014] The observation features generated during the diffusion process by the visual diffusion model are extracted, and the observation features and robot state features are input into the action policy network for action prediction to obtain the action control sequence.

[0015] Optionally, in a first implementation of the first aspect of the present invention, before acquiring the real-time visual image from the current viewpoint, the method further includes:

[0016] Acquire multi-view image samples of robot operation scenarios and annotate the corresponding depth information, 3D point cloud representation and camera pose parameters;

[0017] The multi-view image samples are input into an initial feedforward neural network for optimization training to obtain a geometric estimation model that can output estimated depth information and estimated camera extrinsic parameters at the true scale.

[0018] The feedforward neural network was pre-trained on common 3D vision data.

[0019] Optionally, in a second implementation of the first aspect of the present invention, the visual diffusion model includes a denoising network;

[0020] The observed features generated during the diffusion process by the extracted visual diffusion model include:

[0021] The latent features contained in the last hidden layer of the denoising network in the visual diffusion model are obtained as observation features when the network performs forward computation in a single denoising time step.

[0022] Optionally, in a third implementation of the first aspect of the present invention, the robot state features include the robot's joint angles;

[0023] The step of inputting the observed features and robot state features into the action policy network for action prediction to obtain the action control sequence includes:

[0024] The robot's joint angles are obtained, and a multilayer perceptron is invoked to align the observed features and the robot's joint angles to the same preset dimension for splicing, resulting in a fused feature word sequence.

[0025] The feature word sequence is input into the action policy network, and the action prediction head of the action policy network generates a robot action control sequence corresponding to the current control cycle.

[0026] Optionally, in a fourth implementation of the first aspect of the present invention, the diffusion conditions further include spatiotemporal conditions;

[0027] After estimating the depth image and view-to-view relative pose from the real-time visual image using a geometric estimation model, the method further includes:

[0028] Calculate the viewpoint difference between the current viewpoint and the preset viewpoint based on the relative pose of the viewpoint;

[0029] When the viewpoint difference exceeds a preset threshold, the method further includes performing pose interpolation on the preset viewpoint and the current viewpoint, and rendering frame by frame to obtain a multi-frame interpolated image sequence and a mask sequence.

[0030] The multi-frame interpolated image sequence and the mask sequence are used as the spatiotemporal conditions of the visual diffusion model;

[0031] The step of encoding the coarse-rendered image and calling the visual diffusion model to generate a synthetic image with a preset viewpoint under the current control cycle based on the encoded coarse-rendered image and diffusion conditions includes:

[0032] The coarsely rendered image is encoded to generate a latent space representation;

[0033] The visual diffusion model is invoked to generate a synthetic image from a preset viewpoint under the current control cycle based on the latent space representation, historical time series conditions, and the spatiotemporal conditions.

[0034] Optionally, in a fifth implementation of the first aspect of the invention, the coarse-rendered image is further associated with a projection mask for marking hole regions;

[0035] The step of encoding the coarsely rendered image to generate a latent space representation includes:

[0036] The latent space representation is obtained by encoding the coarse-rendered image and the projection mask using a 3D variational autoencoder.

[0037] A second aspect of the present invention provides a robot cross-view motion control system based on spatiotemporal perspective synthesis, comprising:

[0038] The vision acquisition module is used to acquire real-time visual images from the current viewpoint during the current control cycle.

[0039] The geometric estimation module is used to estimate the depth image and viewpoint relative pose at the current viewpoint by calling the geometric estimation model based on the real-time visual image.

[0040] The point cloud rendering module is used to construct a first point cloud under the current view based on the depth image and camera parameters, and to reproject the first point cloud to a preset view based on the relative pose of the view to obtain a second point cloud; and to generate a coarsely rendered image under the preset view based on the second point cloud.

[0041] The temporal memory module is used to encode the synthetic image generated in the previous control cycle under the preset viewpoint as a historical temporal condition;

[0042] The diffusion synthesis module is used to encode the coarsely rendered image to generate a latent space representation, and call the visual diffusion model to generate a synthesized image with a preset viewpoint under the current control cycle based on the latent space representation and the historical time sequence conditions.

[0043] The action strategy module is used to extract the observation features generated by the visual diffusion model during the diffusion process, and input the observation features and robot state features into the action strategy network to predict actions and obtain the action control sequence.

[0044] A third aspect of the present invention provides a robot with cross-view motion control based on spatiotemporal perspective synthesis, comprising: a robot body, a memory and at least one processor, wherein the memory stores instructions;

[0045] The at least one processor invokes the instructions in the memory to cause the robot to execute the steps of the above-described cross-view motion control method for robots based on spatiotemporal perspective synthesis.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described method for cross-view motion control of a robot based on spatiotemporal perspective synthesis.

[0047] A fifth aspect of the present invention provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the steps of the above-described method for cross-view robot motion control based on spatiotemporal perspective synthesis.

[0048] The technical solution provided by this invention involves acquiring a real-time visual image from the current viewpoint during the current control cycle; estimating the depth image and relative pose of the viewpoint using a geometric estimation model based on the real-time visual image; constructing a first point cloud from the current viewpoint based on the depth image and camera parameters, and reprojecting the first point cloud onto a preset viewpoint based on the relative pose of the viewpoint to obtain a second point cloud; generating a coarsely rendered image from the preset viewpoint based on the second point cloud; encoding the synthetic image from the preset viewpoint generated in the previous control cycle to generate historical temporal conditions; encoding the coarsely rendered image, and calling a visual diffusion model to generate a synthetic image from the preset viewpoint in the current control cycle based on the encoded coarsely rendered image and diffusion conditions, wherein the diffusion conditions include historical temporal conditions; extracting the observation features generated by the visual diffusion model during the diffusion process, and inputting the observation features and robot state features into an action policy network for action prediction to obtain an action control sequence. This method can perform action control on the robot across viewpoints, and can still accurately plan the robot's actions even when changes in viewpoint cause a shift in the distribution of visual features. The system, computer-readable storage medium, computer program product, and robot provided by this invention also solve corresponding technical problems. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a flowchart illustrating the first embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention.

[0051] Figure 2 This is a flowchart illustrating the second embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention.

[0052] Figure 3 This is a schematic diagram of a real-world application scenario of the robot cross-view motion control method based on spatiotemporal perspective synthesis in an embodiment of the present invention.

[0053] Figure 4 This is a closed-loop operation diagram of the robot cross-view motion control method based on spatiotemporal perspective synthesis in an embodiment of the present invention;

[0054] Figure 5 This is a flowchart illustrating the third embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention.

[0055] Figure 6This is a schematic diagram of an embodiment of a robot cross-view motion control system based on spatiotemporal perspective synthesis in this invention. Detailed Implementation

[0056] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of them will be omitted.

[0057] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0058] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0059] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0060] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0061] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0062] See Figure 1 The first embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention includes:

[0063] S101. In the current control cycle, acquire the real-time visual image from the current viewpoint;

[0064] It is understood that the execution subject of this invention can be a robot cross-view motion control system based on spatiotemporal perspective synthesis, or it can be a terminal or a server; the specific implementation is not limited here. This invention will be illustrated using a robot as the execution subject.

[0065] After the robot enters a preset control cycle, it acquires real-time visual images from the current viewpoint through a vision acquisition unit associated with the robot. This vision acquisition unit can be an RGB camera, an RGB-D camera, or other visual sensing devices. The current viewpoint can be the observation viewpoint from any position during the robot's actual operation; this viewpoint does not need to be exactly the same as the viewpoint used during the training phase.

[0066] After acquiring the real-time visual image, the image is used as the environmental observation input for the current control cycle, and then used for subsequent scene geometry estimation and viewpoint transformation processing.

[0067] S102. Based on the real-time visual image, call the geometric estimation model to estimate the depth image and view relative pose under the current viewpoint.

[0068] After acquiring a real-time visual image from the current viewpoint, the image is input into a pre-trained geometric estimation model for inference processing. The geometric estimation model is used to recover the geometric structure information of the scene from the visual image and outputs a depth image from the current viewpoint along with the corresponding camera pose parameters.

[0069] Furthermore, the relative pose relationship between the current viewpoint and the preset viewpoint is calculated based on the camera pose parameters corresponding to the current viewpoint and the preset viewpoint. This relative pose describes the rotational and displacement relationships between the two viewpoints in three-dimensional space, thus providing a geometric transformation basis for subsequent three-dimensional point cloud reprojection.

[0070] S103. Construct the first point cloud under the current view based on the depth image and camera parameters, and reproject the first point cloud to the preset view based on the relative pose of the view to obtain the second point cloud.

[0071] Based on the current viewpoint depth image and camera parameters obtained in step S102, the pixels in the depth image are back-projected to restore the pixel coordinates in the two-dimensional image space to the point coordinates in the three-dimensional space, thereby constructing a three-dimensional point cloud representation under the current viewpoint as the first point cloud.

[0072] Subsequently, based on the relative pose relationship between the current viewpoint and the preset viewpoint, the first point cloud is transformed into spatial coordinates, and the transformed 3D points are reprojected onto the image coordinate system corresponding to the preset viewpoint, thereby obtaining the second point cloud representation under the preset viewpoint. In this way, the scene geometric information observed from the current viewpoint can be transformed into a unified representation under the preset viewpoint.

[0073] S104. Generate a coarsely rendered image from a preset viewpoint based on the second point cloud;

[0074] After obtaining the second point cloud under the preset viewpoint, the second point cloud is projected onto the image plane corresponding to the preset viewpoint, and the corresponding image representation is generated according to the spatial position relationship of the point cloud, thereby obtaining the coarse rendering image under the preset viewpoint.

[0075] It should be noted that due to changes in occlusion relationships between different viewpoints, the coarsely rendered image may contain some unobserved areas or areas with missing information after point cloud reprojection. This coarsely rendered image is used to provide initial structural information of the scene under the preset viewpoint, providing input conditions for subsequent image completion and refinement.

[0076] S105. Encode the synthetic image generated in the previous control cycle under the preset viewpoint to generate historical time sequence conditions.

[0077] In the continuous control of a robot, scene changes between adjacent control cycles typically exhibit a certain degree of temporal continuity. To utilize this temporal information and improve the stability of new perspective image generation, a synthetic image generated in the previous control cycle from a preset perspective is acquired in the current control cycle, and feature encoding processing is performed on this synthetic image.

[0078] The encoded features serve as historical temporal conditions to represent the visual state information of the scene at the previous moment, thereby providing temporal context constraints in the subsequent diffusion generation process to enhance the consistency of the generated results in the temporal dimension.

[0079] S106. Encode the coarse-rendered image and call the visual diffusion model to generate a synthetic image with a preset viewpoint under the current control cycle based on the encoded coarse-rendered image and diffusion conditions.

[0080] In this step, the diffusion conditions include historical temporal conditions. The coarsely rendered image obtained in step S104 is input into the visual encoding module for feature encoding to obtain the corresponding latent space representation. Subsequently, this latent space representation, together with the historical temporal conditions obtained in step S105, is used as the diffusion conditions input into the visual diffusion model.

[0081] The visual diffusion model progressively denoises and reconstructs the latent space representation during diffusion inference. Under the combined constraints of geometric structural information provided by the coarse-rendered image and temporal context information provided by historical time-series conditions, it generates a synthetic image corresponding to the preset viewpoint in the current control cycle. This method can fill in missing regions in the coarse-rendered image, while simultaneously improving the image's visual consistency and detail quality.

[0082] S107. Extract the observation features generated by the visual diffusion model during the diffusion process, input the observation features and robot state features into the action policy network for action prediction, and obtain the action control sequence.

[0083] During the diffusion inference process of the visual diffusion model, observational features describing the current scene state can be extracted from the feature representation of the diffusion model. These observational features reflect the model's comprehensive understanding of the current scene structure and visual information.

[0084] Furthermore, the observed features are fused with the current robot state features, which may include the robot's joint angles, end effector state, or other motion state information. The fused features are input into the motion policy network for motion prediction, thereby outputting a robot motion control sequence corresponding to the current control cycle. The robot executes the corresponding actions according to the motion control sequence to complete the target operation task.

[0085] The embodiments of the present invention can realize cross-viewpoint motion control of robots, and can still achieve the technical effect of accurate motion planning of robots when the visual feature distribution shifts due to changes in viewpoint.

[0086] See Figure 2-4 The second embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention includes:

[0087] It is understood that the execution subject of this invention can be a robot cross-view motion control system based on spatiotemporal perspective synthesis, or it can be a robot terminal; the specific implementation is not limited here. This embodiment of the invention uses a server as an example for illustration.

[0088] S201. Acquire multi-view image samples of the robot operation scene and label the corresponding depth information, 3D point cloud representation and camera pose parameters.

[0089] This embodiment proposes a closed-loop pose control method for a robot. Before performing specific closed-loop control, a geometric estimation model needs to be pre-trained to achieve automatic recognition and conversion of viewpoint information. Training the geometric estimation model first requires preparing training data. This training data can be a dataset containing multi-view images, with each scene typically corresponding to multiple viewpoint images. For each viewpoint image, corresponding geometric annotation information needs to be prepared, thus forming a training sample set with geometric supervision information. This annotation information includes, but is not limited to: a depth map (or depth that can be obtained by rendering a 3D model) under that viewpoint, 3D point cloud representation information, and camera pose parameters; wherein, the camera pose parameters may include camera intrinsic and extrinsic parameters.

[0090] In another implementation, the multi-view image samples are labeled only with the corresponding depth information and camera pose parameters, and the 3D point cloud representation can be calculated based on the depth information and camera pose parameters.

[0091] See Figure 3 In one specific implementation, the robot described in this embodiment can also be used in embodied operation scenarios. The robot includes components such as a robotic arm and grippers. At this time, while acquiring perspective images, visual calibration is performed simultaneously through a camera associated with the robot, or near-real geometric information is obtained through an associated depth device.

[0092] S202. Input multi-view image samples into the initial feedforward neural network for optimization training to obtain a geometric estimation model that can output estimated depth information and estimated camera extrinsic parameters at the real scale.

[0093] First, a pre-trained visual geometry estimation network is selected as the initial base model. This base model can be a feedforward neural network, such as a forward-feedback visual geometry network based on the Transformer structure. This initial feedforward neural network is pre-trained on a general 3D vision dataset (e.g., large-scale scene reconstruction data), and possesses basic geometric understanding capabilities. In a preferred embodiment, the forward-feedback visual geometry network can be a VGGT (Visual Geometry Grounded Transformer).

[0094] In this step, when fine-tuning and optimizing the initial feedforward neural network using multi-view image samples, in each training iteration, the system samples one or more view images from the training dataset and inputs these images into the initial feedforward neural network. After feature extraction and geometric inference, the model outputs multiple geometric quantities, including the depth map, 3D point cloud representation, and camera pose parameters for that view. Several loss functions are constructed, and the model parameters are jointly optimized through the backpropagation algorithm, so that the values ​​of the above loss functions are gradually reduced. This allows the model to gradually learn to recover the depth structure of the scene from single-view or a small number of view images and infer the geometric pose relationship of the camera, resulting in a geometric estimation model that can output estimated depth information at real scale and estimate camera extrinsic parameters.

[0095] The constructed loss functions include, but are not limited to, depth supervision loss function, camera pose supervision loss function, and cross-view 3D geometric consistency constraint loss function. Multi-view geometric supervision training is achieved based on these loss functions.

[0096] In a preferred embodiment, the optimization training process of the geometric estimation model also incorporates scale consistency constraints or robustness training against robotic arm occlusion, enabling the model to stably predict scene depth even when mechanical structures are present, thereby improving the robustness of the solution and reducing failures and errors.

[0097] After the model training is completed, the robot can proceed to the closed-loop reasoning and motion control stage for cross-view motion control during actual operation in this embodiment. In this stage, steps such as S203-S210 are executed within each control cycle; the control cycle can be a fixed time interval, for example, 10ms to 100ms.

[0098] S203. In the current control cycle, acquire the real-time visual image from the current perspective;

[0099] Combination Figure 3 In this embodiment, the current viewpoint can be the test viewpoint during the testing process (such as the first test viewpoint and the second test viewpoint), or it can be the working viewpoint during the actual work process. When the real-time visual image is acquired from the first test viewpoint or the second test viewpoint, its image features will differ from those under the training viewpoint.

[0100] S204. Based on the real-time visual image, call the geometric estimation model to estimate the depth image and view relative pose at the current viewpoint.

[0101] Real-time visual images are input into a geometric estimation model. Based on the trained geometric estimation model, the depth information of the real-time visual image at the current viewpoint is estimated, resulting in a depth image at the current viewpoint. The geometric estimation model takes a single frame or multiple frames of viewpoint images as input and outputs a depth map, camera pose parameters, and the corresponding 3D structural representation.

[0102] In this embodiment, the camera's pose parameters in 3D space under the current viewpoint are also estimated. Based on the camera pose parameters under the current viewpoint and the pose parameters of a preset viewpoint, the relative pose transformation matrix between the current viewpoint and the preset viewpoint is calculated, thereby estimating the relative pose of the current viewpoint. The preset viewpoint mentioned in this embodiment can be the training viewpoint used during robot training.

[0103] S205. Construct the first point cloud under the current view based on the depth image and camera parameters, and reproject the first point cloud to the preset view based on the relative pose of the view to obtain the second point cloud.

[0104] Based on the depth image of the current viewpoint estimated in S204 and the camera's intrinsic parameters, back projection is performed to obtain the first point cloud under the current viewpoint. Based on the relative pose transformation matrix estimated in S204, the pixels in the depth image are back projected through the camera's intrinsic parameter matrix, and the first point cloud is reprojected to the preset viewpoint. The two-dimensional pixel coordinates are restored to three-dimensional spatial coordinates, thereby constructing the three-dimensional point cloud inference result under the preset viewpoint and obtaining the second point cloud.

[0105] S206. Generate a coarsely rendered image from a preset viewpoint based on the second point cloud;

[0106] After obtaining the second point cloud, the second point cloud is projected onto the image plane corresponding to the preset viewpoint based on the point cloud projection rendering method or the depth testing method based on Z-buffer, and a coarse rendering image under the preset viewpoint is generated based on the second point cloud.

[0107] Because changes in viewpoint cause objects in the scene to occlude each other to varying degrees, the generated second point cloud will contain some holes after the viewpoint change. These holes will also appear in the coarse rendering image. Therefore, in generating the coarse rendering image at the preset viewpoint, a projection mask is also generated to mark the hole areas in the coarse rendering image.

[0108] S207. Calculate the view difference between the current view and the preset view. When the view difference exceeds the preset threshold, perform pose conversion interpolation to obtain a multi-frame interpolated image sequence and a mask sequence, and generate spatiotemporal conditions.

[0109] When the perspectives of two cameras differ significantly, a direct perspective transition will result in numerous hole areas, producing substantial hole repair artifacts that can negatively impact subsequent diffusion generation and even motion control processes. Therefore, it is necessary to construct several intermediate virtual perspectives between the two viewpoints to gradually transition between them, thereby reducing the large-area holes generated by a single reprojection.

[0110] Based on this, in a preferred embodiment, the viewpoint difference is first calculated based on the relative pose between the current viewpoint and the preset viewpoint in the aforementioned steps, and it is determined whether the viewpoint difference exceeds a preset viewpoint threshold. If it does, the pose transformation from the current viewpoint to the preset viewpoint is interpolated, and multi-frame interpolated image sequences and mask sequences are obtained by rendering frame by frame. The multi-frame interpolated image sequences and mask sequences are encoded to generate the spatiotemporal conditions of the visual diffusion model.

[0111] The viewpoint difference can be calculated based on the difference in rotation matrix and displacement vector between the camera pose at a preset viewpoint and the camera pose at the current viewpoint. For example, it can be measured by calculating the difference in rotation angle and the Euclidean distance between the camera centers. The pose interpolation can be performed by applying spherical linear interpolation (SLERP) to the rotation component of the camera pose and by linear interpolating the translation vector, thereby generating multiple intermediate virtual camera poses.

[0112] If the viewpoint difference does not exceed the preset viewpoint threshold, the pose conversion interpolation step in S207 is not required. Instead, the process proceeds directly to S208 to obtain historical timing conditions. Subsequently, the coarse-rendered image projection mask and historical timing conditions are used to perform specific diffusion generation.

[0113] S208. Encode the synthetic image generated in the previous control cycle under the preset viewpoint to generate historical time sequence conditions.

[0114] Acquire the synthetic image generated in the previous control cycle from the preset perspective (i.e., the inference perspective image in the previous control cycle), encode it as a historical time-series condition, and inject the historical time-series condition into the diffusion condition through cross attention.

[0115] S209. Encode the coarse-rendered image and call the visual diffusion model to generate a synthetic image with a preset viewpoint under the current control cycle based on the encoded coarse-rendered image and diffusion conditions.

[0116] The coarsely rendered image from step S206 is input into a 3D-Variational Autoencoder (3D-VAE) for encoding, and the projection mask used to identify holes is used as an auxiliary condition and input into the visual diffusion model together with the encoded latent space representation to obtain the latent space representation.

[0117] In this embodiment, the diffusion conditions include the latent space representation and historical time series conditions. When the difference between the two camera viewpoints is too large, the diffusion conditions also include the spatiotemporal conditions generated in S207.

[0118] The coarsely rendered image and diffusion conditions are input into the visual diffusion model for denoising and to generate a new perspective image.

[0119] In a preferred embodiment, the specific steps of generating new perspective images can be completed in the background through parallel processing, or the specific new perspective images can be omitted, and only the latent space representation in the hidden layer of the visual diffusion model can be extracted as the representation information for scene observation.

[0120] S210. When the denoising network in the visual diffusion model performs forward computation in a single denoising time step, the latent features contained in the last hidden layer are used as observation features. The observation features and robot state features are input into the action policy network for action prediction to obtain the action control sequence.

[0121] Since the focus of this embodiment is not on generating new perspective images, but on predicting actions based on the model's understanding of the current perspective, directly using the diffusion model to decode the new perspective image would result in long temporal inference time and additional overhead from the decoding step. Therefore, we use the features of a one-step denoised diffusion model as the observation input for the action strategy, that is, we only perform a single forward computation of the denoised network at a fixed time step of the diffusion model, without performing the complete multi-step diffusion sampling process.

[0122] Specifically, this embodiment only performs one denoising step. The latent features contained in the last hidden layer of the visual diffusion model during the denoising process are extracted as observation features. These observation features are then concatenated and fused with the current robot state features, and input into the action policy network for action prediction to obtain the action control sequence. The observation features can be... The tensor form, in which Indicates altitude, Indicates width, Indicates the number of dimensions.

[0123] In one specific implementation, the robot state features are the robot's joint angles. The fusion of the observed features and the current robot state features includes: activating a multilayer perceptron to align the observed features and the robot's joint angles to the same preset dimension for fusion, resulting in a fused feature token sequence; subsequently, the feature token sequence is input into an action policy network, and the action prediction head of the action policy network generates a robot action control sequence corresponding to the current control cycle. The action policy network can employ a Transformer policy network or a policy network structure based on a multilayer perceptron.

[0124] See Figure 4 In the diagram, 'o' represents observation, 'a' represents action, and 't' represents time. The upper-level flow in the diagram indicates that observations from unseen perspectives can lead to action deviations and task failures. The lower-level flow, through a perspective synthesis process, combines observations from unseen perspectives with historical references to achieve consistent action prediction and successfully complete the task. Based on this, the solution in this embodiment can achieve perspective generalization. Visual features learned under a fixed training perspective can still achieve relatively stable visual control when distribution shifts occur in new perspectives. It reduces action deviations in closed-loop execution and avoids automatic execution failures. In testing and / or actual working scenarios, it does not rely on calibration, making it more convenient and efficient to use.

[0125] See Figure 5 The third embodiment of the robot cross-view motion control method based on spatiotemporal perspective synthesis in this invention includes:

[0126] When the preset view is the training view and the current view is the test view, the specific processing flow is as follows: After the process starts, it first enters S501;

[0127] S501, Load training viewpoint reference frames And initiate a closed-loop control cycle;

[0128] in, This represents the initial training viewpoint image.

[0129] S502. Acquire test view frame of the current control cycle. ;

[0130] In this step, This represents the image from the test perspective.

[0131] S503, Estimation , Building point clouds and render and ;

[0132] Call the trained geometric estimation model, based on the test view image. Estimated depth map and relative pose Subsequently, based on the camera's internal parameters... Depth map Test view image Back projection of point cloud for test view Then through relative pose Transform the point cloud from the test viewpoint. By switching to the training perspective, we obtain the inference point cloud from the training perspective. And based on reasoning point clouds Rendering yields a coarse rendering image of the inference from the training perspective. With projection mask .

[0133] S504. Determine whether the angle difference exceeds the preset range. If yes, proceed to S505; otherwise, proceed to S506.

[0134] S505. Generate an interpolated image sequence and a mask sequence as diffusion conditions;

[0135] If the viewpoint difference exceeds a preset range, it is considered that the angle difference between the current test viewpoint and the training viewpoint is large. Pose interpolation is performed on the training and test viewpoints, and multi-frame interpolated image sequences and mask sequences are obtained by rendering frame by frame. These are used as the spatiotemporal conditions for the video diffusion model, and the inference coarse rendering image is then used. Projection mask The multi-frame interpolated image sequence and mask sequence obtained from the rendering are used as diffusion conditions;

[0136] S506, Direct Use , As a diffusion condition;

[0137] If the angle difference does not exceed the preset range, it is considered that the angle difference between the current test view and the training view is small. To improve efficiency, the coarse rendering image of inference can be used directly. and projection mask As a diffusion condition.

[0138] S507: Read historical memory conditions and perform diffusion synthesis, output multi-layer features and construct... ;

[0139] In the closed-loop control process, this embodiment caches the training perspective results generated in the previous control cycle and encodes these results as historical memory conditions. In this step, the historical memory conditions are read, and based on the diffusion conditions in S505 or S506, they are injected into the diffusion generation process through cross-attention. Interpolation is performed on the multi-layer features (such as upsampling layer features) from the final denoising step of the diffusion model, and the features are concatenated by channel to obtain the scene observation latent features. .

[0140] When the current closed-loop control process is the first control cycle, and the training perspective results generated in the previous control cycle are not present in the cache, then the historical memory condition is not used or the training perspective is used as the historical memory condition.

[0141] S508, Policy Network Input and Output the action and execute it;

[0142] Obtain the real-time robot body state characteristics in the current control cycle of the closed-loop control process. To observe latent features of the scene Robot body state characteristics The system integrates and employs a Transformer architecture to output closed-loop control actions. The output can support either block-based or sequential action output.

[0143] S509. Cache the results generated in this cycle and write them to the memory module;

[0144] The generation result of the current cycle written into the memory module can be used as a historical memory condition similar to that described in S507 in the next cycle to provide memory conditions for the action control of the next cycle.

[0145] S510: Should we continue to the next control cycle? If yes, return to S502; otherwise, end.

[0146] For specific implementation details in this embodiment, please refer to the content of the aforementioned second embodiment, which will not be repeated here.

[0147] The solution in this embodiment of the invention can achieve viewpoint generalization. When the visual features learned under a fixed training viewpoint produce a distribution shift under a new viewpoint, relatively stable visual control can still be achieved. In closed-loop execution, the deviation of actions is reduced, and automatic execution failure is avoided.

[0148] The above describes the robot cross-view motion control method based on spatiotemporal perspective synthesis in the embodiments of the present invention. The following describes the robot cross-view motion control system based on spatiotemporal perspective synthesis in the embodiments of the present invention. (See reference...) Figure 6 One embodiment of the robot cross-view motion control system based on spatiotemporal perspective synthesis in this invention includes:

[0149] The vision acquisition module 601 is used to acquire real-time visual images from the current viewpoint during the current control cycle.

[0150] The geometric estimation module 602 is used to estimate the depth image and viewpoint relative pose under the current viewpoint by calling the geometric estimation model based on the real-time visual image;

[0151] The point cloud rendering module 603 is used to construct a first point cloud under the current view based on the depth image and camera parameters, and to reproject the first point cloud to a preset view based on the relative pose of the view to obtain a second point cloud; and to generate a coarse rendering image under the preset view based on the second point cloud.

[0152] The temporal memory module 604 is used to encode the synthetic image generated in the previous control cycle under the preset viewpoint as a historical temporal condition;

[0153] The diffusion synthesis module 605 is used to encode the coarsely rendered image to generate a latent space representation, and call the visual diffusion model to generate a synthesized image with a preset viewpoint under the current control cycle based on the latent space representation and the historical time sequence conditions.

[0154] The action strategy module 606 is used to extract the observation features generated by the visual diffusion model during the diffusion process, and input the observation features and robot state features into the action strategy network to predict actions and obtain an action control sequence.

[0155] The embodiments of the present invention can realize cross-viewpoint motion control of robots, and can still accurately plan the robot's motion when the visual feature distribution shifts due to changes in viewpoint.

[0156] In another embodiment of this application, a model training module is also included, which is used to acquire multi-view image samples of the robot operation scene and annotate the corresponding depth information, 3D point cloud representation and camera pose parameters.

[0157] The multi-view image samples are input into an initial feedforward neural network for optimization training to obtain a geometric estimation model that can output estimated depth information and estimated camera extrinsic parameters at the true scale.

[0158] The feedforward neural network was pre-trained on common 3D vision data.

[0159] In another embodiment of this application, the visual diffusion model includes a denoising network; the action strategy module 606 is specifically used to obtain the latent features contained in the last hidden layer of the denoising network in the visual diffusion model as observation features when performing forward computation of a single denoising time step.

[0160] In another embodiment of this application, the robot state features include the robot's joint angles; the motion strategy module 606 is further configured to:

[0161] The robot's joint angles are obtained, and a multilayer perceptron is invoked to align the observed features and the robot's joint angles to the same preset dimension for splicing, resulting in a fused feature word sequence.

[0162] The feature word sequence is input into the action policy network, and the action prediction head of the action policy network generates a robot action control sequence corresponding to the current control cycle.

[0163] In another embodiment of this application, the diffusion conditions further include spatiotemporal conditions;

[0164] The timing memory module 604 is further used for:

[0165] Calculate the viewpoint difference between the current viewpoint and the preset viewpoint based on the relative pose of the viewpoint;

[0166] When the viewpoint difference exceeds a preset threshold, the method further includes performing pose interpolation on the preset viewpoint and the current viewpoint, and rendering frame by frame to obtain a multi-frame interpolated image sequence and a mask sequence.

[0167] The multi-frame interpolated image sequence and the mask sequence are used as the spatiotemporal conditions of the visual diffusion model;

[0168] The diffusion synthesis module 605 is further used for:

[0169] The coarsely rendered image is encoded to generate a latent space representation;

[0170] The visual diffusion model is invoked to generate a synthetic image from a preset viewpoint under the current control cycle based on the latent space representation, historical time series conditions, and spatiotemporal conditions.

[0171] In another embodiment of this application, the coarse-rendered image is also associated with a projection mask for marking the hole regions;

[0172] The diffusion synthesis module 605 is also used for:

[0173] The step of encoding the coarsely rendered image to generate a latent space representation includes:

[0174] The latent space representation is obtained by encoding the coarse-rendered image and the projection mask using a 3D variational autoencoder.

[0175] The embodiments of the present invention can realize cross-viewpoint motion control of robots. When the visual feature distribution shifts due to changes in viewpoint, the invention can infer the visual information of the current scene under a identifiable preset viewpoint based on real-time visual image information, and perform motion planning and control of the robot, thereby improving the viewpoint generalization ability and reducing motion deviation or motion control failure.

[0176] Based on the same inventive concept, embodiments of this specification also provide a robot capable of cross-view motion control based on spatiotemporal perspective synthesis. The robot includes: a robot body, a memory, and at least one processor. The memory stores instructions. The at least one processor calls the instructions in the memory to cause the robot to perform the steps of the robot cross-view motion control method based on spatiotemporal perspective synthesis as described in the above embodiments.

[0177] In one specific implementation, the robot body is a robotic arm with grippers, and the robot also includes a visual acquisition unit such as a camera.

[0178] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 , Figure 2 or Figure 5 The method shown.

[0179] The present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the robot cross-view motion control method based on spatiotemporal perspective synthesis as described in any of the above embodiments.

[0180] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0181] accomplish Figure 1 , Figure 2 or Figure 5The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0182] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0183] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0184] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0185] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0186] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for robot cross-view action control based on spatio-temporal view synthesis, characterized in that, include: In the current control cycle, acquire real-time visual images from the current perspective; Based on the real-time visual image, the geometric estimation model is invoked to estimate the depth image and view-relative pose at the current viewpoint. The first point cloud is constructed based on the depth image and camera parameters at the current viewpoint, and the first point cloud is reprojected onto the preset viewpoint based on the relative pose of the viewpoint to obtain the second point cloud. A coarsely rendered image from a preset viewpoint is generated based on the second point cloud; Encode the synthetic image generated in the previous control cycle from the preset viewpoint to generate historical time sequence conditions; Calculate the viewpoint difference between the current viewpoint and the preset viewpoint based on the relative pose of the viewpoint; When the viewpoint difference exceeds a preset threshold, the method further includes performing pose interpolation on the preset viewpoint and the current viewpoint, and rendering frame by frame to obtain a multi-frame interpolated image sequence and a mask sequence. The multi-frame interpolated image sequence and the mask sequence are used as the spatiotemporal conditions of the visual diffusion model; The coarsely rendered image is encoded to generate a latent space representation, and a visual diffusion model is invoked to generate a synthetic image with a preset viewpoint under the current control cycle based on the latent space representation, historical time series conditions, and the spatiotemporal conditions. The observation features generated during the diffusion process by the visual diffusion model are extracted, and the observation features and robot state features are input into the action policy network for action prediction to obtain the action control sequence.

2. The robot cross-view motion control method based on spatiotemporal perspective synthesis according to claim 1, characterized in that, Before acquiring the real-time visual image from the current viewpoint, the method further includes: Acquire multi-view image samples of robot operation scenarios and annotate the corresponding depth information, 3D point cloud representation and camera pose parameters; The multi-view image samples are input into an initial feedforward neural network for optimization training to obtain a geometric estimation model that can output estimated depth information and estimated camera extrinsic parameters at the true scale. The feedforward neural network was pre-trained on common 3D vision data.

3. The robot cross-view motion control method based on spatiotemporal perspective synthesis according to claim 1, characterized in that, The visual diffusion model includes a denoising network; The observed features generated during the diffusion process by the extracted visual diffusion model include: The latent features contained in the last hidden layer of the denoising network in the visual diffusion model are obtained as observation features when the network performs forward computation in a single denoising time step.

4. The robot cross-view motion control method based on spatiotemporal perspective synthesis according to claim 1, characterized in that, The robot's state characteristics include the robot's joint angles; The step of inputting the observed features and robot state features into the action policy network for action prediction to obtain the action control sequence includes: The robot's joint angles are obtained, and a multilayer perceptron is invoked to align the observed features and the robot's joint angles to the same preset dimension for splicing, resulting in a fused feature word sequence. The feature word sequence is input into the action policy network, and the action prediction head of the action policy network generates a robot action control sequence corresponding to the current control cycle.

5. The robot cross-view motion control method based on spatiotemporal perspective synthesis according to claim 1, characterized in that, The coarse-rendered image is also associated with a projection mask used to mark the hole areas; The step of encoding the coarsely rendered image to generate a latent space representation includes: The latent space representation is obtained by encoding the coarse-rendered image and the projection mask using a 3D variational autoencoder.

6. A robot cross-view motion control system based on spatiotemporal perspective synthesis, characterized in that, The robot cross-view motion control system based on spatiotemporal perspective synthesis includes: The vision acquisition module is used to acquire real-time visual images from the current viewpoint during the current control cycle. The geometric estimation module is used to estimate the depth image and viewpoint relative pose at the current viewpoint by calling the geometric estimation model based on the real-time visual image. The point cloud rendering module is used to construct a first point cloud under the current view based on the depth image and camera parameters, and to reproject the first point cloud to a preset view based on the relative pose of the view to obtain a second point cloud; and to generate a coarsely rendered image under the preset view based on the second point cloud. The temporal memory module is used to encode the synthetic image generated in the previous control cycle under a preset viewpoint as a historical temporal condition; and to calculate the viewpoint difference between the current viewpoint and the preset viewpoint based on the relative pose of the viewpoint; when the viewpoint difference exceeds a preset threshold, it further includes performing pose interpolation on the preset viewpoint and the current viewpoint, and rendering frame by frame to obtain a multi-frame interpolated image sequence and a mask sequence; the multi-frame interpolated image sequence and the mask sequence are used as the spatiotemporal conditions of the visual diffusion model; The diffusion synthesis module is used to encode the coarsely rendered image to generate a latent space representation, and call the visual diffusion model to generate a synthesized image with a preset viewpoint under the current control cycle based on the latent space representation, historical time series conditions and the spatiotemporal conditions. The action strategy module is used to extract the observation features generated by the visual diffusion model during the diffusion process, and input the observation features and robot state features into the action strategy network to predict actions and obtain the action control sequence.

7. A robot capable of cross-viewpoint motion control based on spatiotemporal perspective synthesis, characterized in that, The robot includes: a robot body, a memory, and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the robot to perform the steps of the robot cross-view motion control method based on spatiotemporal perspective synthesis as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program / instructions thereon, characterized in that, When the program / instruction is executed by the processor, it implements the steps of the robot cross-view motion control method based on spatiotemporal perspective synthesis as described in any one of claims 1-5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the robot cross-view motion control method based on spatiotemporal perspective synthesis as described in any one of claims 1-5 are implemented.

Images