A robot field of view video generation method and system

Abstract

The embodiment of the application provides a robot field-of-view video generation method and system, relates to the technical field of robot visual perception, and obtains multi-view image data, and estimates modeling parameters by using a three-dimensional estimation model. The three-dimensional rendering model is trained by using the modeling parameters, then the camera trajectory is generated according to the motion characteristic parameters of the target robot. By inputting the camera trajectory into the three-dimensional rendering model, the image of each camera pose in the camera trajectory is rendered based on the three-dimensional rendering model, and a new view image sequence is obtained, so that the field-of-view video is generated according to the new view image sequence. The method can estimate the camera parameters and the scene point cloud simultaneously by using the visual Transformer type three-dimensional estimation model, and the image is rendered by using the three-dimensional rendering model in combination with the camera trajectory related to the motion law of the robot, so that the complete process from the multi-view image input to the robot field-of-view video output is realized, and the generation efficiency and the generation quality of the field-of-view video are improved.

Description

Technical Field

[0001] This application relates to the field of robot vision perception technology, and in particular to a method and system for generating robot field-of-view videos. Background Technology

[0002] Tasks such as robot visual perception, embodied intelligence training, imitation learning, reinforcement learning, and robot head-view modeling require the collection of a large amount of video data that is consistent with or close to the robot's own perspective. For example, when a robot performs tasks in a real environment, the video sequences obtained by its head camera can serve as an important data source for subsequent visual policy training, world model training, action prediction, navigation perception, or human-computer interaction tasks.

[0003] However, in actual product development and process flow, there are significant limitations to directly using the robot itself for large-scale data collection. On the one hand, robot-based data collection is costly, requiring physical equipment, space, personnel, and debugging time. On the other hand, when robots collect data in real-world environments, they are easily affected by factors such as range of motion, motion safety, hardware wear and tear, synchronization errors, sensor installation location, and difficulty in reproducing the environment, resulting in low data collection efficiency, poor repeatability, and insufficient coverage.

[0004] Therefore, generating visual videos simulating robot head trajectories from existing multi-view image data, without or with minimal robot involvement, has significant engineering value and application implications. For example, multi-view images can be obtained using fixed camera arrays, handheld cameras, or surround-view imaging devices. New perspective images can then be generated through 3D reconstruction or neural rendering methods to simulate the actual observation perspective of the robot head during movement. These new perspective images are then integrated into visual field video data. However, the generation efficiency and quality of this visual field video data are relatively low, making it unsuitable for subsequent robot training. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and system for generating robot field-of-view video to solve the problems of low generation efficiency and low generation quality of field-of-view video data.

[0006] According to a first aspect of this application, a method for generating robot field-of-view video is provided, the method comprising: Acquire multi-view image data, which includes multiple view images belonging to the same target at the same time. Based on the multi-view image data, a 3D estimation model is used to estimate the modeling parameters. The 3D estimation model is an image processing model built based on a visual transformer. The modeling parameters include scene point clouds and camera parameters. The modeling parameters are used to train a 3D rendering model, which is used to initialize a 3D representation with the scene point cloud and to perform training and optimization with the multi-view image data and the camera parameters as supervision signals. A camera trajectory is generated based on the motion characteristic parameters of the target robot, and the camera trajectory includes multiple camera poses; The camera trajectory is input into the 3D rendering model to render images of each camera pose in the camera trajectory based on the 3D rendering model, thereby obtaining a new perspective image sequence. A field-of-view video is generated based on the new perspective image sequence.

[0007] In some embodiments, acquiring multi-view image data includes: Obtain downstream task configuration information, including the frame rate of the field-of-view video; Calculate the target time based on the downstream task configuration information; An image acquisition command is generated according to the target time, and the image acquisition command is sent to a multi-view image acquisition device so that the multi-view image acquisition device acquires view images according to the frame rate; the multi-view image acquisition device includes at least one of a surround camera array, a multi-camera synchronous acquisition system, and a handheld multi-view acquisition device. Receive multiple perspective images acquired by the perspective image acquisition device in response to the image acquisition command; Based on the target time, multiple viewpoint images are grouped to obtain the multi-viewpoint image data.

[0008] In some embodiments, based on the multi-view image data, a three-dimensional estimation model is used to estimate modeling parameters, including: Multiple viewpoint images belonging to the same target time are input into the three-dimensional estimation model; The visual information of the viewpoint image is converted into sequential data by the visual transformer in the three-dimensional estimation model; The sequence data is subjected to deep feature extraction and global relation modeling based on self-attention mechanism and feedforward network to generate initial point cloud and initial parameters; Preprocessing is performed on the initial point cloud and the initial parameters according to preset preprocessing terms to obtain the scene point cloud and the camera parameters; wherein, the preset preprocessing terms include at least one of the following for the initial point cloud: removing low confidence points, removing outliers based on depth range, scale normalization, and coordinate system unification; the preset preprocessing terms also include at least one of the following for the initial parameters: extrinsic parameter smoothing, extrinsic parameter constraint optimization, and intrinsic parameter consistency correction.

[0009] In some embodiments, the 3D rendering model is a 3D Gaussian rendering model; training the 3D rendering model using the modeling parameters includes: Based on the scene point cloud, a three-dimensional Gaussian representation is initialized for each three-dimensional point. The three-dimensional Gaussian representation includes Gaussian parameters. The Gaussian parameters are used to characterize at least one of the position, scale, rotation, color, opacity, and spherical harmonic coefficients of the ellipsoid corresponding to the three-dimensional Gaussian representation. A three-dimensional Gaussian differentiable renderer is used to project the three-dimensional Gaussian representation onto a two-dimensional image plane to obtain a projected image; Based on the camera parameters, the projected image is rasterized and rendered to generate a predicted image from multiple viewpoints. The Gaussian parameters of the 3D Gaussian representation are iteratively optimized end-to-end based on the loss function, which includes at least one of pixel reconstruction loss, structural similarity loss, perceptual loss, depth consistency loss, transparency or sparsity regularization term, geometric smoothness constraint, and multi-view color consistency constraint. When the decrease in the training loss value calculated based on the loss function is less than or equal to the magnitude threshold, or when the number of iterations reaches a preset threshold, the Gaussian parameters corresponding to the three-dimensional Gaussian representation are saved to obtain the three-dimensional Gaussian rendering model corresponding to the current frame.

[0010] In some embodiments, end-to-end iterative optimization of the Gaussian parameters of the three-dimensional Gaussian representation based on a loss function includes: Randomly sample camera viewpoints from the multi-view image data; Using the current 3D Gaussian rendering model and the camera parameters corresponding to the camera viewpoint, perform differentiable rasterization rendering to generate a predicted image from the camera viewpoint. The training loss value is calculated according to the loss function by comparing the predicted image and the viewpoint image. The Gaussian parameters of the 3D Gaussian representation are updated based on the backpropagation of the training loss value.

[0011] In some embodiments, generating a camera trajectory based on the motion characteristic parameters of the target robot includes: The motion characteristic parameters of the target robot are obtained, including the target robot's field of vision height, head turning angle range, motion speed, rotation speed, task-related observation target area, and trajectory smoothing constraints. Determine the initial pose point based on the camera parameters at the target time; Multiple simulated pose points are generated based on the initial pose point and the motion feature parameters. The simulated pose points include simulated camera position and simulated camera pose. The multiple simulated pose points are combined to form the camera trajectory.

[0012] In some embodiments, multiple simulated pose points are generated based on the initial pose point and the motion feature parameters, including: Using the initial pose point as a reference, the target region is delineated according to the motion feature parameters; Based on the trajectory smoothing constraint, multiple simulated pose points are randomly generated within the target area, wherein the multiple simulated pose points are continuously distributed in terms of camera position and / or camera attitude. or, The target robot's simulated motion is obtained, and the simulated motion includes at least one of horizontal rotation, vertical rotation, shaking, target following motion, and patrol motion; Using the motion feature parameters and the initial pose point as initial conditions, multiple simulated pose points are generated according to the action to be simulated.

[0013] In some embodiments, generating a field-of-view video based on the new perspective image sequence includes: The new perspective image sequence is reordered according to the camera trajectory to obtain an initial rendered video, which includes multiple initial rendered video frames. The initial rendered video frame is input into the diffusion model to perform image quality enhancement on the initial rendered video frame. The diffusion model is a neural network model obtained by supervised training using the initial rendered video frame as sample input and the target real video frame as quality constraint. The image quality enhancement includes at least one of the following: improving image sharpness, eliminating rendering artifacts, filling in occluded areas, enhancing texture details, improving consistency between video frames, and adjusting visual distribution. Obtain the field-of-view video output by the diffusion model.

[0014] In some embodiments, the method further includes: Extract the target's real video frame from the multi-view image data; The initial rendered video frame is input into the diffusion model to obtain a training augmented video, which includes multiple training augmented video frames. The model parameters of the diffusion model are iteratively adjusted by comparing the trained augmented video frames with the target real video frames.

[0015] According to a second aspect of this application, a robot vision video generation system is provided, the system comprising: The data acquisition module is used to acquire multi-view image data, which includes multiple view images belonging to the same target at the same time. The parameter estimation module is used to estimate modeling parameters based on the multi-view image data using a 3D estimation model. The 3D estimation model is an image processing model built based on a visual transformer. The modeling parameters include scene point clouds and camera parameters. The Gaussian rendering module is used to train a 3D rendering model using the modeling parameters. The 3D rendering model is used to initialize a 3D representation using the scene point cloud and to perform training and optimization using the multi-view image data and the camera parameters as supervision signals. The trajectory generation module is used to generate a camera trajectory based on the motion characteristic parameters of the target robot, and the camera trajectory includes multiple camera poses. The image rendering module is used to input the camera trajectory into the three-dimensional rendering model, so as to perform image rendering on each camera pose in the camera trajectory based on the three-dimensional rendering model to obtain a new perspective image sequence. The video generation module is used to generate a field-of-view video based on the new perspective image sequence.

[0016] By employing the above technical solutions, embodiments of this application provide a method and system for generating robot field-of-view videos. This method acquires multi-view image data and uses a 3D estimation model to estimate modeling parameters. The modeling parameters are then used to train a 3D rendering model, and camera trajectories are generated based on the motion characteristic parameters of the target robot. By inputting the camera trajectories into the 3D rendering model, image rendering is performed on each camera pose within the camera trajectory based on the 3D rendering model, resulting in a new viewpoint image sequence. This new viewpoint image sequence is then used to generate a field-of-view video. The method can simultaneously estimate camera parameters and scene point clouds using a visual Transformer-type 3D estimation model, and uses a 3D rendering model combined with camera trajectories related to robot motion laws for image rendering. This achieves an end-to-end video generation process from multi-view image input to robot field-of-view video output, improving the generation efficiency and quality of the field-of-view video.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart of the robot field-of-view video generation method provided in the embodiments of this application; Figure 2This is a schematic diagram of the multi-view image data acquisition process provided in the embodiments of this application; Figure 3 This is a schematic diagram of the modeling parameter estimation process provided in an embodiment of this application; Figure 4 This is a schematic diagram of the field-of-view video generation process provided in an embodiment of this application; Figure 5 This is a schematic diagram of the diffusion model training process provided in an embodiment of this application; Figure 6 This is a schematic diagram of the robot vision video generation system provided in an embodiment of this application. Detailed Implementation

[0019] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0020] In this embodiment, a robot refers to a mobile or humanoid intelligent machine body equipped with visual sensors and used for embodied intelligence, imitation learning, or visual strategy training. In some embodiments, the robot includes a mechanical system, a drive system, a perception system, and a control system. The mechanical system may include a support mechanism, an execution mechanism, and a locomotion mechanism. Taking a humanoid robot as an example, the support mechanism may include a base, waist, arms, etc., for supporting the entire robot body and integrated components and enabling a wide range of movement. The execution mechanism may include end effectors such as robot wrists and hands for performing fine operations, such as grasping, welding, and spraying. The locomotion mechanism may include wheels, feet, etc., for enabling the robot to move.

[0021] A drive system provides power to a mechanical system, receives commands from the control system, and drives the movement of various joints. For example, a robot's drive system can use electric drive methods such as servo motors and stepper motors to achieve precise control and rapid response. Drive systems can also include hydraulic drives for heavy-duty applications and pneumatic drives with simpler structures.

[0022] A perception system is used to perceive the robot itself and its external environment, and can include internal and external sensors. Internal sensors detect the robot's own state to maintain stable movement, such as joint angles, velocity, and acceleration. External sensors acquire environmental information, allowing the robot to perceive the external world. Examples include visual sensors such as RGB-D cameras and depth cameras, which can be used for localization, navigation, and object recognition. In addition, external sensors can also include force, tactile, and auditory sensors.

[0023] The control system is responsible for processing sensor information, making decisions and planning, and issuing action commands. The control system can support various control methods, including centralized and distributed control. Centralized control refers to the entire robot system being controlled by a single computer or processor; distributed control refers to a control method where multiple computers work together to control the system.

[0024] In tasks such as robot visual perception, embodied intelligence training, imitation learning, reinforcement learning, and robot head vision modeling, it is often necessary to collect a large amount of data that is consistent with or close to the robot's own perspective. For example, when a robot performs a task in a real environment, the video sequences obtained by its head camera can serve as an important data source for subsequent visual policy training, world model training, action prediction, navigation perception, or human-computer interaction tasks.

[0025] However, in actual product development and process flow, directly using the robot itself for large-scale data collection has significant limitations. On the one hand, robot-based data collection is costly, requiring physical equipment, space, personnel, and debugging time. On the other hand, when robots collect data in real-world environments, they are easily affected by factors such as range of motion, motion safety, hardware wear and tear, synchronization errors, sensor installation location, and difficulty in reproducing the environment, resulting in low data collection efficiency, poor repeatability, and insufficient coverage. Therefore, relying solely on robot-based data collection is insufficient to meet the needs of constructing large-scale training data.

[0026] For example, robot vision data acquisition requires mounting cameras on the robot's head or end effector and acquiring video data from the corresponding perspective by actually controlling the robot's movement. This method of video data acquisition requires actual robot equipment, resulting in high acquisition costs. Furthermore, the acquisition process is limited by the robot's kinematic structure, joint constraints, safety strategies, and site conditions, making trajectory reproduction difficult when acquiring data from the same scene multiple times. If a large amount of head-view trajectory data is required, the acquisition cycle is long, hardware wear and tear is significant, and it is difficult to acquire data in dangerous, narrow, complex, or inconvenient robot deployment scenarios.

[0027] In some embodiments, visual videos simulating the trajectory of a robot's head can be generated from existing multi-view image data without or with minimal robot involvement, based on methods such as novel perspective synthesis, 3D reconstruction, and data augmentation.

[0028] For example, in scenarios without body-based image acquisition, multi-view images can be obtained through fixed camera arrays, handheld cameras, surround shooting devices, or other acquisition methods. These images can then be used to generate new view images through 3D reconstruction or neural rendering. However, the viewpoints of these images are not necessarily equivalent to the actual observation viewpoints of the robot's head during movement. Furthermore, when converting multi-view static or short-term image data into new trajectory videos that conform to the robot's head movement patterns, the unknown or inaccurate camera parameters between the multi-view images and the unstable estimation of the scene's 3D structure can affect subsequent rendering quality. The robot's head trajectory needs to meet certain requirements for motion continuity, viewpoint rationality, and task relevance. Processing individual frames independently can easily cause inter-frame flickering, texture drift, or geometric inconsistencies. Moreover, the rendered video may not directly meet the downstream training requirements in terms of detail, clarity, and realism.

[0029] In some embodiments, novel perspective synthesis can also be achieved through multi-view geometric 3D reconstruction, neural radiation fields, and 3D Gaussian representation methods. Among these, multi-view geometric methods rely on feature matching, structured bundle adjustment, and dense reconstruction, making the process complex and sensitive to texture loss, dynamic objects, and lighting changes. While neural radiation field methods offer good quality novel perspective synthesis, their training and rendering costs are high during frame-by-frame scene modeling, making them unsuitable for large-scale frame-level processing requirements.

[0030] The 3D Gaussian representation method has advantages in efficiency, but if it is directly trained by relying on point clouds and camera parameters estimated from multi-view images, the rendering results may still have problems such as blurriness, holes, artifacts or color inconsistencies due to input geometric errors, sparse viewpoints, missing occluded areas, etc.

[0031] For robot vision training, generated videos not only need to have high realism in single-frame vision, but also need to meet the stability and usability requirements of continuous video. If the rendered video has problems such as noise, artifacts, texture blurring, edge drift, and local structural distortion, it may cause the subsequent model to learn the wrong visual distribution, thereby affecting the policy training, visual representation learning, or world model training results. Therefore, the generation efficiency and quality of the vision video data generated by the above video synthesis methods are relatively low.

[0032] To address the issues of low generation efficiency and quality of field-of-view video data, some embodiments of this application provide a robot field-of-view video generation method. This method can simultaneously estimate camera parameters and scene point clouds using 3D estimation models such as the Visual Transformer, and uses a 3D rendering model combined with camera trajectories related to robot motion patterns for image rendering. This achieves an end-to-end video generation process from multi-view image input to robot field-of-view video output, thereby improving the generation efficiency and quality of field-of-view video.

[0033] The method can be applied to robot control systems with vision capabilities and can be deployed on GPU servers, robot data processing platforms, or offline data generation platforms. Its input is a multi-view image sequence, and its output is a video sequence of the robot's head field of vision. The method can be applied to robot training data generation processes and can also be integrated into dataset construction systems, simulation data augmentation systems, or embodied intelligent model training platforms.

[0034] The method can also be applied to electronic devices that establish communication connections with robots and have data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application embodiment uses a control system as the executing entity of the method. It should be understood that the method can also be applied to other types of executing entities, which are not illustrated in this application embodiment. Figure 1 As shown, the method includes: S101. Acquire multi-view image data.

[0035] To generate video from the robot's field of view, the robot control system can first acquire multi-view image data using a vision sensor. This multi-view image data includes multiple view images belonging to the same target time. The view images within the multi-view image data can be grouped according to their acquisition time, thus view images belonging to the same acquisition time can form an image group. The multi-view image data can include image groups corresponding to multiple acquisition times. The target time is any one of these multiple acquisition times. That is, among the multiple acquisition times, the acquisition time used for the current frame video generation process can be called the target time.

[0036] For each frame in the video or scene sequence to be processed, multiple viewpoint images corresponding to that frame can be acquired. For example, each frame includes eight images from different viewpoints, which can be obtained by a surround camera array, a multi-camera synchronous acquisition system, a handheld multi-view acquisition device, or other multi-view acquisition devices. The eight images can be used to characterize the multi-view observation results of the target scene at the same or approximately the same time, and each image can have known or unknown initial camera parameters.

[0037] Depending on the number and configuration of the visual sensors included in the multi-view acquisition device, the number of viewpoint images belonging to the same target at any given time can be dynamically set as needed. For example, in some examples, eight images are used per frame, while in others, four, six, ten, or more images may be used per frame. As long as multi-view scene observation can be provided, it can be applied to the generation of field-of-view video.

[0038] In some embodiments, multi-view image data can also be acquired according to downstream task requirements. For example... Figure 2 As shown, when acquiring multi-view image data, downstream task configuration information can be obtained first, including the frame rate of the field-of-view video.

[0039] The target time is then calculated based on the downstream task configuration information, and an image acquisition command is generated according to the target time. This command is then sent to the multi-view image acquisition device, enabling it to acquire view images at the specified frame rate. The system then receives multiple view images acquired by the view image acquisition device in response to the image acquisition command and groups these images based on the target time to obtain multi-view image data.

[0040] For example, by reading the downstream task configuration information, if the required field-of-view video frame rate for the downstream training task is 30fps, then based on the frame rate of the field-of-view video, the image sampling interval can be determined to be 1000 / 30 = 33ms. Simultaneously, the current time T is obtained, and multiple target times are calculated based on the sampling interval, such as T1 = T+33, T2 = T1+33, etc. Then, an image acquisition command is generated based on the calculated target times and sent to a surround camera array, a multi-camera synchronous acquisition system, a handheld multi-view acquisition device, or other multi-view image acquisition devices, triggering the multi-view image acquisition devices to acquire multi-view images according to the target times or sampling frame rates. Finally, eight view images belonging to the same target time are grouped together to form multi-view image data.

[0041] S102. Based on multi-view image data, use a three-dimensional estimation model to estimate the modeling parameters.

[0042] After acquiring multi-view image data, modeling parameters can be estimated using a 3D estimation model based on the multi-view image data. The 3D estimation model is an image processing model built based on a visual transformer, such as a Visual Geometry Grounded Transformer (VGGT) or similar visual transformer model.

[0043] In some embodiments, in addition to the visual Transformer model, other multi-view 3D estimation models, Structure from Motion (SfM) methods, Multi-View Stereo (MVS) methods, depth estimation models, or learning-based camera pose estimation models can be used for modeling parameter estimation.

[0044] The modeling parameters estimated using a 3D estimation model can include scene point clouds and camera parameters. The scene point cloud refers to the collection of all 3D points in the scene, representing a digital sample of the surface of a real-world object. The scene point cloud can include 3D coordinates, color information, and point cloud confidence scores.

[0045] Camera parameters can include intrinsic and extrinsic parameters. Intrinsic parameters describe the camera's physical and optical properties, such as the lens focal length and the center position of the image sensor. The lens focal length and the center position of the image sensor determine the image position of a point in space. Extrinsic parameters describe the camera's position and orientation in the world coordinate system, such as translation vectors and rotation matrices. Extrinsic parameters determine the camera's viewing angle of the scene.

[0046] In some embodiments, such as Figure 3 As shown, when estimating modeling parameters using a 3D estimation model based on multi-view image data, multiple view images belonging to the same target time can first be input into the 3D estimation model. The visual information of the view images is then converted into sequential data by the visual transformer in the 3D estimation model. Next, deep feature extraction and global relationship modeling are performed on the sequential data based on a self-attention mechanism and a feedforward network to generate initial point clouds and initial parameters.

[0047] For example, the input data for a 3D estimation model consists of 8 viewpoint images corresponding to the same frame. After inputting the multi-viewpoint image data into the 3D estimation model, the model can first perform feature extraction, dividing the 8 input viewpoint images into multiple image patches and converting them into initial feature vectors.

[0048] Then, information interaction and fusion are carried out. That is, the initial feature vector is fed into multiple consecutive ViT encoder modules. Within the ViT encoder module, the self-attention mechanism is used to process the information association within a single image, and the cross-view attention mechanism is used to transfer and aggregate information among the eight images to obtain the final features after association, transfer and aggregation.

[0049] The final features processed by the encoder are sent to different task heads, such as regression head or classification head, for task output. The task head is then used to generate scene point clouds and camera parameters, thus obtaining the initial point cloud and initial parameters.

[0050] After obtaining the initial point cloud and initial parameters, preprocessing is performed on the initial point cloud and initial parameters according to preset preprocessing items to obtain the scene point cloud and camera parameters. The preset preprocessing items include at least one of the following for the initial point cloud: removal of low-confidence points, removal of outliers based on depth range, scale normalization, and coordinate system unification. The preset preprocessing items also include at least one of the following for the initial parameters: extrinsic parameter smoothing, extrinsic parameter constraint optimization, and intrinsic parameter consistency correction.

[0051] For example, when estimating scene point clouds and camera parameters based on a visual Transformer model, eight images corresponding to each frame can be input into a visual Transformer-based 3D estimation model. The 3D estimation model can output sparse, semi-dense, or dense point clouds of the current scene; camera intrinsic parameters corresponding to each input image; camera extrinsic parameters corresponding to each input image; in addition, the 3D estimation model can also output the confidence score, depth map, matching relationship, or visibility information for each 3D point.

[0052] After obtaining the output of the 3D estimation model, the initial point cloud and initial parameters can be preprocessed. For the initial point cloud, low-confidence points can be removed, outliers can be eliminated based on depth range, and the point cloud can be scaled and the coordinate system unified. For the camera extrinsic parameters in the initial parameters, smoothing or constraint optimization can be performed; for the camera intrinsic parameters in the initial parameters, consistency correction can be performed. After the above processing, scene point cloud (initial geometric data) and camera parameters suitable for subsequent 3D Gaussian training can be obtained.

[0053] S103. Train the 3D rendering model using modeling parameters.

[0054] After obtaining the modeling parameters through the 3D estimation model, the 3D rendering model can be trained based on the generated scene point cloud and camera parameters. That is, the 3D rendering model is trained using the modeling parameters. The 3D rendering model can adopt a 3D Gaussian rendering model, or it can adopt a neural radiation field, volume rendering model, explicit mesh rendering, point cloud neural rendering, or a hybrid representation method.

[0055] Because the 3D Gaussian rendering model has significant advantages in execution efficiency and engineering implementation, it is used in some embodiments. This 3D rendering model is then used to initialize a 3D representation using scene point clouds and to perform training and optimization using multi-view image data and camera parameters as supervisory signals.

[0056] In some embodiments, when training a 3D rendering model using modeling parameters, a 3D Gaussian representation can be initialized for each 3D point based on the scene point cloud. The 3D Gaussian representation includes Gaussian parameters; these parameters characterize at least one of the following: the position, scale, rotation, color, opacity, and spherical harmonic coefficients of the ellipsoid corresponding to the 3D Gaussian representation.

[0057] For example, when training a 3D Gaussian rendering model based on scene point clouds and camera parameters, a 3D Gaussian representation can be initialized based on the scene point cloud data estimated by the 3D estimation model, that is, a 3D Gaussian ellipsoid is initialized for each 3D point. Each 3D Gaussian ellipsoid can include parameters such as position, scale, rotation, color, opacity, and optional spherical harmonics. Among the Gaussian parameters, position can be initialized to point cloud coordinates; scale can be initialized based on point cloud density or neighborhood distance, such as isotropic small scales; rotation can be initialized to unit quaternions; color can be taken from point cloud color or corresponding image pixel color; opacity can be initialized to a fixed value (e.g., 0.5) or weighted by confidence; spherical harmonics can be used for view-dependent color modeling and initialized to zero or random small values.

[0058] After initializing the 3D Gaussian representation, a 3D Gaussian differentiable renderer is used to project the 3D Gaussian representation onto a 2D image plane to obtain a projected image. The projected image is then rasterized and rendered according to camera parameters to generate predicted images from multiple viewpoints. Finally, end-to-end iterative optimization of the Gaussian parameters of the 3D Gaussian representation is performed based on a loss function. This loss function includes at least one of the following: pixel reconstruction loss, structural similarity loss, perceptual loss, depth consistency loss, transparency or sparsity regularization term, geometric smoothness constraint, and multi-viewpoint color consistency constraint.

[0059] For example, after initializing the 3D Gaussian representation, a differentiable rendering pipeline can be established. This involves using a 3D Gaussian differentiable renderer to project the 3D Gaussian onto a 2D image plane, and for each input viewpoint, performing rasterization rendering based on its camera intrinsic and extrinsic parameters to generate a predicted image for that viewpoint. This predicted image can contain information such as RGB color and depth.

[0060] Next, a loss function is constructed. For each input viewpoint, various losses between the rendered image and the real input image are calculated. The loss function can be a weighted sum of losses such as pixel reconstruction loss, structural similarity loss, perceptual loss, depth consistency loss, opacity or sparsity regularization term, geometric smoothness constraint, and multi-view color consistency constraint. Among them, pixel reconstruction loss includes L1 or L2 loss; structural similarity loss includes SSIM loss; perceptual loss includes feature distance based on pre-trained VGG networks; depth consistency loss; opacity sparsity regularization term can encourage opacity to tend to 0 or 1; geometric smoothness constraint is a penalty for adjacent Gaussian positions or scale differences; and multi-view color consistency loss is used to determine whether the color projection of the same 3D point should be consistent under different viewpoints.

[0061] After constructing the loss function, iterative optimization can be performed based on the loss function, such as using the Adam optimizer or similar gradient descent methods to perform end-to-end iterative optimization of Gaussian parameters such as position, scale, rotation, color, opacity, and spherical harmonic coefficients of the 3D Gaussian.

[0062] In some embodiments, when performing end-to-end iterative optimization of the Gaussian parameters of the 3D Gaussian representation based on a loss function, camera views can be randomly sampled from multi-view image data first. Differentiable rasterization rendering is then performed using the current 3D Gaussian rendering model and the camera parameters corresponding to the camera views to generate a predicted image from the camera viewpoint. Then, by comparing the predicted image and the viewpoint image, the training loss value is calculated according to the loss function, and the Gaussian parameters of the 3D Gaussian representation are updated through backpropagation based on the training loss value.

[0063] For example, in each iteration, a batch of input viewpoints can be randomly sampled, and images of those viewpoints can be rendered. Then, the total loss can be calculated based on the loss function, and backpropagation can be performed to update the Gaussian parameters. During training, the learning rate can also be dynamically adjusted, and adaptive density control can be incorporated, such as splitting or pruning Gaussians based on gradient sparsity.

[0064] During each iteration of optimization, a convergence judgment is required. When the decrease in the training loss value calculated based on the loss function is less than or equal to the magnitude threshold, or when the number of iterations reaches the preset number threshold, the Gaussian parameters corresponding to the 3D Gaussian representation are saved to obtain the 3D Gaussian rendering model corresponding to the current frame.

[0065] For example, eight input images and their corresponding camera intrinsics and extrinsic parameters can be used as supervision signals to train or optimize a 3D Gaussian model, making the rendering result of the 3D Gaussian model as consistent as possible with the real image from the input viewpoint. Training can be stopped when the loss value no longer decreases significantly on the validation set, or when a preset number of iterations is reached, and the final set of 3D Gaussian parameters can be saved as the 3D Gaussian rendering model for the current frame. Therefore, through iterative optimization training, the 3D Gaussian rendering model for the current frame can be obtained. The 3D Gaussian rendering model can generate corresponding new viewpoint images given any reasonable camera pose and camera intrinsics.

[0066] S104. Generate camera trajectory based on the motion characteristic parameters of the target robot.

[0067] After or simultaneously with training the 3D rendering model, motion characteristic parameters of the target robot can also be obtained. The target robot refers to a robot used for tasks such as visual perception, embodied intelligence training, imitation learning, reinforcement learning, and head-view modeling. The motion characteristic parameters refer to a set of parameters that characterize and constrain the robot's actual movements and visual features. These parameters may include at least one of the following: the target robot's field of vision height, head turning angle range, movement speed, rotation speed, task-related observation target area, and trajectory smoothing constraints.

[0068] After obtaining the motion characteristic parameters of the target robot, a camera trajectory is generated based on these parameters. This is achieved by generating a new perspective trajectory using a pre-set head trajectory generation algorithm. The head trajectory generation algorithm can be a rule-based algorithm, a robot kinematic model, a real trajectory statistical model, a learned trajectory generation model, or a task-driven trajectory planning method, used to generate the camera trajectory. The camera trajectory includes multiple camera poses.

[0069] In some embodiments, when generating a camera trajectory based on the motion feature parameters of the target robot, the motion feature parameters of the target robot can be obtained first, and an initial pose point can be determined based on the camera parameters at the target time. Then, multiple simulated pose points are generated based on the initial pose point and the motion feature parameters, wherein the simulated pose points include simulated camera position and simulated camera attitude. Finally, the multiple simulated pose points are combined into a camera trajectory.

[0070] For example, to generate a new perspective trajectory for the robot's head, a new set of camera trajectories can be generated based on the motion characteristics of the target robot's head camera. The head trajectory can be obtained by a head trajectory generation algorithm, whose inputs include motion feature parameters, namely, the installation height of the target robot's head camera; the range of head yaw, pitch, and roll angles; the robot's motion speed or head rotation speed; the task-related observation target area; and trajectory smoothing constraints. By executing the head trajectory generation algorithm, the head trajectory can be automatically calculated, which simulates the changes in the field of view that the robot's head camera may observe in the current scene. The generated head trajectory can then include a series of camera poses, each pose including camera position and camera orientation.

[0071] In some embodiments, the head trajectory can be a small-range translational and rotational trajectory around the target area. That is, when generating multiple simulated pose points based on the initial pose point and motion feature parameters, the target area is first delineated according to the motion feature parameters using the initial pose point as a reference, and then multiple simulated pose points are randomly generated within the target area based on trajectory smoothing constraints. These multiple simulated pose points are continuously distributed in terms of camera position and / or camera attitude.

[0072] For example, if the world coordinate origin is represented as T = (0, 0, 0), then the initial pose point is the default initial position of the head. Assuming the robot is facing the target, it is located at (0, 0, 1) meters away, which is 1 meter away from the target in the positive Z-axis direction. The camera orientation always maintains its gaze at the origin (0, 0, 0), meaning that no matter how the head translates or rotates, the center of its line of sight always points towards the target.

[0073] By defining trajectory parameters that include two simultaneous motions, each conforming to a sine or cosine law to ensure smoothness, a head-shaking motion is simulated by a horizontal rotation (yaw angle) of ±5°. A forward / backward translation of ±5 cm simulates head-raising and lowering for observation. The camera pose corresponding to the time series t can then be generated using the following formula. Assume a total duration of 4 seconds, a frame rate of 30fps, and a total of 120 frames.

[0074] We can first calculate the rotation amount, representing the change in camera orientation. Then, the camera's rotation on the Y-axis (yaw angle) changes according to a sine law:

[0075] Next, calculate the translation amount, representing the change in camera position. The camera position then changes along the Z-axis according to a cosine law:

[0076] Then, the final camera pose is synthesized, i.e., position: P(t) = (0,0,1 + offset_z(t)); attitude: the camera gazes at the origin (0,0,0), where the up direction is (0,1,0). In this way, when the robot's head rotates horizontally, its line of sight is always locked on the tabletop object.

[0077] Therefore, multi-frame pose data can be generated according to the above calculation method, as shown in Table 1: Table 1. Examples of the first 5 frames of data generated for the trajectory;

[0078] In some embodiments, the head trajectory can be generated by simulating the robot's head rotation. That is, when generating multiple simulated pose points based on an initial pose point and motion feature parameters, the target robot's simulated action can be obtained first. The simulated action includes at least one of horizontal rotation, vertical rotation, jitter, target following action, and patrol action. Then, using the motion feature parameters and the initial pose point as initial conditions, multiple simulated pose points are generated according to the simulated action.

[0079] S105. Input the camera trajectory into the 3D rendering model, and perform image rendering on each camera pose in the camera trajectory based on the 3D rendering model to obtain a new perspective image sequence.

[0080] After generating the camera trajectory, a trained 3D Gaussian model can be used to render the image under that camera trajectory, i.e., generating a new perspective video based on the 3D Gaussian rendering model. During the generation of the new perspective video, the camera trajectory can be input into the 3D rendering model to render images of each camera pose within the camera trajectory based on the 3D rendering model, resulting in a sequence of new perspective images.

[0081] For example, the head trajectory generated in the above example can be input into the trained 3D Gaussian rendering model to render each camera pose in the head trajectory, thereby obtaining the corresponding new perspective image sequence.

[0082] S106. Generate a field-of-view video based on the new perspective image sequence.

[0083] By performing the above steps on each frame of the original video sequence, multiple new perspective image sequences corresponding to the robot's head trajectory can be obtained. These multiple new perspective image sequences can then be stitched together to generate a field-of-view video, such as... Figure 4 As shown.

[0084] The video formed by stitching together multiple new perspective image sequences corresponds to the observation path of the target robot's head camera in terms of geometric perspective. However, due to the limited number of multi-view inputs, errors in point cloud estimation, and sparse training data for the 3D Gaussian model, the video may have problems such as local blurring, holes, artifacts, texture discontinuities, or insufficient realism. Therefore, the video synthesized by performing the above processing on each frame can be called the preliminary rendering video.

[0085] In some embodiments, a diffusion model can also be used to enhance the image quality of the initially rendered video to obtain an enhanced field-of-view video, which can then be used as head field-of-view data for subsequent training of the robot. A diffusion model is a generative artificial intelligence model inspired by the molecular diffusion phenomenon in physics. It can learn how to generate high-quality data from pure noise by mimicking diffusion phenomena in the physical world.

[0086] The diffusion model can be divided into two core stages: the forward process and the backward process. The forward process, also known as the noise-adding process, adds a small amount of Gaussian noise to a sharp input image at each step t. After a sufficient number of noise-adding steps, all the structure and details of the original image are completely destroyed, eventually becoming a completely random Gaussian noise map. The noise distribution is called the prior distribution.

[0087] The reverse process, also known as the denoising process, trains a neural network based on the U-Net architecture to predictively remove noise from a purely noisy image step by step, ultimately restoring a clear image.

[0088] Neural network models, through deep learning, can learn the characteristics of noise added at each step. During inference, starting from pure noise, they gradually remove noise in reverse order based on the learned patterns. Furthermore, by introducing text descriptions, category labels, or low-quality images as conditional inputs, the model's denoising process can be guided to generate images or videos that meet specific requirements.

[0089] When generating a view video based on a new perspective image sequence, the new perspective image sequence can be reordered according to the camera trajectory to obtain an initial rendering video, wherein the initial rendering video includes multiple initial rendering video frames.

[0090] The initial rendered video frames are then input into the diffusion model to perform image quality enhancement on them, thereby obtaining the visual field video output by the diffusion model. The diffusion model is a neural network model obtained through supervised training, using the initial rendered video frames as input samples and the target real video frames as quality constraints. The image quality enhancements that the diffusion model can perform include at least one of the following: improving image sharpness, eliminating rendering artifacts, filling in occluded areas, enhancing texture details, improving inter-frame consistency, and adjusting visual distribution.

[0091] For example, inputting the initial rendered video into a trained or fine-tuned diffusion model can yield a quality-enhanced video. The enhanced video can have higher resolution, better texture detail, fewer rendering artifacts, and more stable inter-frame consistency. This video can be used as robot head vision data for subsequent robot vision model training, policy learning, data augmentation, simulation testing, or other downstream tasks.

[0092] It should be noted that the diffusion model can be replaced with other video enhancement models, super-resolution models, GAN models, Transformer video inpainting models, or temporal consistency-based image enhancement models. Furthermore, point cloud estimation and 3D Gaussian training can be performed independently for each frame, or sliding window, multi-frame joint modeling, temporal consistency constraints, or global scene models can be used to further improve video stability and processing efficiency.

[0093] By applying the technical solutions of the above embodiments, the robot vision video generation method described in the above embodiments takes multi-view images corresponding to each moment or each frame as input, obtains point cloud data of the current scene and intrinsic and extrinsic parameters of multiple cameras through a visual Transformer-type 3D estimation model, then trains a 3D rendering model based on the point cloud and camera parameters, and generates a new perspective trajectory that conforms to the robot's head movement law through a head trajectory generation algorithm, and renders the new perspective video along the trajectory using the trained 3D rendering model, finally, to address the problems of blurring, artifacts and lack of realism that may exist in the initial rendered video, a diffusion model finely tuned based on the video or related videos of the same scene is used to enhance the video quality, resulting in video data that can be directly used as the robot's head vision. The method can use multi-view images to recover the 3D expression of a single frame scene, achieve efficient new perspective rendering through a 3D model, combine a head trajectory generation algorithm to simulate the robot's observation path, and use a diffusion model to improve the quality of the rendered video, thereby replacing or reducing the need for real robot body acquisition.

[0094] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a robot vision video generation method. The difference between this method and the above embodiments is that it can fine-tune the diffusion model based on a preliminary rendered video, such as... Figure 5 As shown, the method includes: S201. Extract the target's real video frames from multi-view image data; S202. Input the initial rendered video frames into the diffusion model to obtain the trained and enhanced video. S203. By comparing the enhanced video frames with the target real video frames during training, the model parameters of the diffusion model are iteratively adjusted.

[0095] After generating a new perspective video based on a 3D Gaussian rendering model, the diffusion model can be fine-tuned based on the initial rendered video. This can be achieved by extracting target real video frames from multi-view image data and inputting the initial rendered video frames into the diffusion model to obtain a training augmented video. The training augmented video includes multiple training augmented video frames. Then, by comparing the training augmented video frames with the target real video frames, the model parameters of the diffusion model are iteratively adjusted.

[0096] For example, to improve the visual quality of the initial rendered video, the diffusion model can be fine-tuned based on the obtained rendered video. The conditional inputs of the diffusion model are the initial rendered video frames and the target real video frames. The training objectives of the diffusion model include: improving image sharpness, eliminating rendering artifacts, filling in occluded areas, enhancing texture details, improving the consistency between video frames, and making the generated video more closely resemble the visual distribution of videos captured by a real robot head camera.

[0097] By using the initial rendered video as low-quality input and the target real video frames as quality constraints, the diffusion model can be trained in a supervised or weakly supervised manner, thereby obtaining a diffusion model that can improve image quality.

[0098] By applying the technical solutions of the above embodiments, the robot vision video generation method described above can take the conversion of multi-view images into robot head trajectory videos as a clear objective, automatically obtain point clouds and camera parameters through a visual Transformer model, and reduce calibration dependence. Furthermore, it achieves new perspective rendering of controllable trajectories through 3D Gaussian representation. Then, based on the initial rendered video, the diffusion model is fine-tuned, and the diffusion model is used to compensate for insufficient rendering quality, thereby allowing the final output to be directly used for robot training data production.

[0099] In some embodiments, as a specific implementation of the robot vision video generation method described in the above embodiments, some embodiments of this application also provide a robot vision video generation system, such as... Figure 6 As shown, the system includes: The data acquisition module is used to acquire multi-view image data, which includes multiple view images belonging to the same target at the same time. The parameter estimation module is used to estimate modeling parameters based on the multi-view image data using a 3D estimation model. The 3D estimation model is an image processing model built based on a visual transformer. The modeling parameters include scene point clouds and camera parameters. The Gaussian rendering module is used to train a 3D rendering model using the modeling parameters. The 3D rendering model is used to initialize a 3D representation using the scene point cloud and to perform training and optimization using the multi-view image data and the camera parameters as supervision signals. The trajectory generation module is used to generate a camera trajectory based on the motion characteristic parameters of the target robot, and the camera trajectory includes multiple camera poses. The image rendering module is used to input the camera trajectory into the three-dimensional rendering model, so as to perform image rendering on each camera pose in the camera trajectory based on the three-dimensional rendering model to obtain a new perspective image sequence. The video generation module is used to generate a field-of-view video based on the new perspective image sequence.

[0100] By applying the technical solutions of the above embodiments, the robot vision video generation system provided in the above embodiments can acquire multi-view image data through the data acquisition module and estimate modeling parameters using a 3D estimation model through the parameter estimation module. The Gaussian rendering module then trains a 3D rendering model using the modeling parameters, and the trajectory generation module generates camera trajectories based on the motion characteristic parameters of the target robot. The image rendering module inputs the camera trajectory into the 3D rendering model, and performs image rendering on each camera pose in the camera trajectory based on the 3D rendering model to obtain a new viewpoint image sequence. This allows the video generation module to generate a vision video based on the new viewpoint image sequence. The system can use a visual Transformer-type 3D estimation model to simultaneously estimate camera parameters and scene point clouds, and uses a 3D rendering model combined with camera trajectories related to robot motion laws for image rendering, realizing an end-to-end video generation process from multi-view image input to robot vision video output, improving the generation efficiency and quality of vision videos.

[0101] It should be noted that other corresponding descriptions of the functional units involved in the robot vision video generation system provided in this application embodiment can be found in the corresponding descriptions in the robot vision video generation method provided in the above embodiments, and will not be repeated here.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A method for generating robot field-of-view video, characterized in that, The method includes: Acquire multi-view image data, which includes multiple view images belonging to the same target at the same time. Based on the multi-view image data, a 3D estimation model is used to estimate the modeling parameters. The 3D estimation model is an image processing model built based on a visual transformer. The modeling parameters include scene point clouds and camera parameters. The modeling parameters are used to train a 3D rendering model, which is used to initialize a 3D representation with the scene point cloud and to perform training and optimization with the multi-view image data and the camera parameters as supervision signals. A camera trajectory is generated based on the motion characteristic parameters of the target robot, and the camera trajectory includes multiple camera poses; The camera trajectory is input into the 3D rendering model to render images of each camera pose in the camera trajectory based on the 3D rendering model, thereby obtaining a new perspective image sequence. A field-of-view video is generated based on the new perspective image sequence.

2. The method according to claim 1, characterized in that, Acquire multi-view image data, including: Obtain downstream task configuration information, including the frame rate of the field-of-view video; Calculate the target time based on the downstream task configuration information; An image acquisition command is generated according to the target time, and the image acquisition command is sent to a multi-view image acquisition device so that the multi-view image acquisition device acquires view images according to the frame rate; the multi-view image acquisition device includes at least one of a surround camera array, a multi-camera synchronous acquisition system, and a handheld multi-view acquisition device. Receive multiple perspective images acquired by the perspective image acquisition device in response to the image acquisition command; Based on the target time, multiple viewpoint images are grouped to obtain the multi-viewpoint image data.

3. The method according to claim 1, characterized in that, Based on the multi-view image data, a three-dimensional estimation model is used to estimate the modeling parameters, including: Multiple viewpoint images belonging to the same target time are input into the three-dimensional estimation model; The visual information of the viewpoint image is converted into sequential data by the visual transformer in the three-dimensional estimation model; The sequence data is subjected to deep feature extraction and global relation modeling based on self-attention mechanism and feedforward network to generate initial point cloud and initial parameters; Preprocessing is performed on the initial point cloud and the initial parameters according to preset preprocessing terms to obtain the scene point cloud and the camera parameters; wherein, the preset preprocessing terms include at least one of the following for the initial point cloud: removing low confidence points, removing outliers based on depth range, scale normalization, and coordinate system unification; the preset preprocessing terms also include at least one of the following for the initial parameters: extrinsic parameter smoothing, extrinsic parameter constraint optimization, and intrinsic parameter consistency correction.

4. The method according to claim 1, characterized in that, The 3D rendering model is a 3D Gaussian rendering model; training the 3D rendering model using the modeling parameters includes: Based on the scene point cloud, a three-dimensional Gaussian representation is initialized for each three-dimensional point. The three-dimensional Gaussian representation includes Gaussian parameters. The Gaussian parameters are used to characterize at least one of the position, scale, rotation, color, opacity, and spherical harmonic coefficients of the ellipsoid corresponding to the three-dimensional Gaussian representation. A three-dimensional Gaussian differentiable renderer is used to project the three-dimensional Gaussian representation onto a two-dimensional image plane to obtain a projected image; Based on the camera parameters, the projected image is rasterized and rendered to generate a predicted image from multiple viewpoints. The Gaussian parameters of the 3D Gaussian representation are iteratively optimized end-to-end based on the loss function, which includes at least one of pixel reconstruction loss, structural similarity loss, perceptual loss, depth consistency loss, transparency or sparsity regularization term, geometric smoothness constraint, and multi-view color consistency constraint. When the decrease in the training loss value calculated based on the loss function is less than or equal to the magnitude threshold, or when the number of iterations reaches a preset threshold, the Gaussian parameters corresponding to the three-dimensional Gaussian representation are saved to obtain the three-dimensional Gaussian rendering model corresponding to the current frame.

5. The method according to claim 4, characterized in that, End-to-end iterative optimization of the Gaussian parameters of the 3D Gaussian representation based on the loss function includes: Randomly sample camera viewpoints from the multi-view image data; Using the current 3D Gaussian rendering model and the camera parameters corresponding to the camera viewpoint, perform differentiable rasterization rendering to generate a predicted image from the camera viewpoint. The training loss value is calculated according to the loss function by comparing the predicted image and the viewpoint image. The Gaussian parameters of the 3D Gaussian representation are updated based on the backpropagation of the training loss value.

6. The method according to claim 1, characterized in that, Generate camera trajectories based on the motion characteristic parameters of the target robot, including: The motion characteristic parameters of the target robot are obtained, including the target robot's field of vision height, head turning angle range, motion speed, rotation speed, task-related observation target area, and trajectory smoothing constraints. Determine the initial pose point based on the camera parameters at the target time; Multiple simulated pose points are generated based on the initial pose point and the motion feature parameters. The simulated pose points include simulated camera position and simulated camera pose. The multiple simulated pose points are combined to form the camera trajectory.

7. The method according to claim 6, characterized in that, Multiple simulated pose points are generated based on the initial pose point and the motion feature parameters, including: Using the initial pose point as a reference, the target region is delineated according to the motion feature parameters; Based on the trajectory smoothing constraint, multiple simulated pose points are randomly generated within the target area, wherein the multiple simulated pose points are continuously distributed in terms of camera position and / or camera attitude. or, The target robot's simulated motion is obtained, and the simulated motion includes at least one of horizontal rotation, vertical rotation, shaking, target following motion, and patrol motion; Using the motion feature parameters and the initial pose point as initial conditions, multiple simulated pose points are generated according to the action to be simulated.

8. The method according to claim 1, characterized in that, Generate a field-of-view video based on the new perspective image sequence, including: The new perspective image sequence is reordered according to the camera trajectory to obtain an initial rendered video, which includes multiple initial rendered video frames. The initial rendered video frame is input into the diffusion model to perform image quality enhancement on the initial rendered video frame. The diffusion model is a neural network model obtained by supervised training using the initial rendered video frame as sample input and the target real video frame as quality constraint. The image quality enhancement includes at least one of the following: improving image sharpness, eliminating rendering artifacts, filling in occluded areas, enhancing texture details, improving consistency between video frames, and adjusting visual distribution. Obtain the field-of-view video output by the diffusion model.

9. The method according to claim 8, characterized in that, The method further includes: Extract the target's real video frame from the multi-view image data; The initial rendered video frame is input into the diffusion model to obtain a training augmented video, which includes multiple training augmented video frames. The model parameters of the diffusion model are iteratively adjusted by comparing the trained augmented video frames with the target real video frames.

10. A robot vision video generation system, characterized in that, The system includes: The data acquisition module is used to acquire multi-view image data, which includes multiple view images belonging to the same target at the same time. The parameter estimation module is used to estimate modeling parameters based on the multi-view image data using a 3D estimation model. The 3D estimation model is an image processing model built based on a visual transformer. The modeling parameters include scene point clouds and camera parameters. The Gaussian rendering module is used to train a 3D rendering model using the modeling parameters. The 3D rendering model is used to initialize a 3D representation using the scene point cloud and to perform training and optimization using the multi-view image data and the camera parameters as supervision signals. The trajectory generation module is used to generate a camera trajectory based on the motion characteristic parameters of the target robot, and the camera trajectory includes multiple camera poses. The image rendering module is used to input the camera trajectory into the three-dimensional rendering model, so as to perform image rendering on each camera pose in the camera trajectory based on the three-dimensional rendering model to obtain a new perspective image sequence. The video generation module is used to generate a field-of-view video based on the new perspective image sequence.

Images