A target-driven mobile grasping control method for multi-scene environment

Abstract

The application relates to a target-driven mobile grabbing control method for a multi-scene environment, which comprises the following steps: inputting an RGB-D sampling sequence of multiple sub-scenes, and initializing a 3D Gaussian model of the whole scene by using a sparse 3D point cloud with semantics; obtaining an approximate pose of a target by inquiring a coarse-grained Gaussian, guiding a mobile chassis to approach the target, and reducing the range of pose estimation on the target to a corresponding sub-scene; in the mobile process, optimizing a fine-grained Gaussian by using diffusion-based depth completion, and obtaining an accurate pose of the target; performing Gaussian editing on the fine-grained Gaussian model, and obtaining a robot operation terminal pose by Gaussian repair; and controlling a robot manipulator to complete corresponding actions according to a robot operation starting pose and the operation terminal pose. Compared with the prior art, the application combines rough scene priori and fine pose estimation, can realize efficient and accurate multi-scene mobile grabbing control, and ensures that the robot can accurately perform a task in a complex multi-scene environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot operation control, and in particular to a target-driven mobile grasping control method for a multi-scene environment. BACKGROUND

[0002] A service robot mainly consists of a mobile chassis and a robotic arm. In daily tasks, the service robot usually needs to navigate and perform fine operations in a multi-sub-scene environment. However, existing technologies mostly focus on complex tasks in a single scene or simple tasks across scenes, making it difficult to effectively handle complex tasks in multiple sub-scenes and facing the challenge of efficiently representing and coordinating the mobile chassis and the robotic arm in collaboration in multiple sub-scenes.

[0003] Traditional methods rely on single-view observation or a large amount of computing resources to complete tasks, which is inefficient and inaccurate when dealing with cross-scene tasks. Specifically, traditional robot systems mainly rely on complete modeling and planning of a single scene when dealing with complex tasks. These systems can complete tasks such as grasping, carrying, and stacking by performing fine identification and operation on objects in a single scene. However, when moving between multiple different scenes and performing a series of complex operations, existing methods often fall short and cannot obtain the precise pose of the starting and ending points of the robot operation.

[0004] Some research attempts to achieve cross-scene task execution by constructing 3D scene graphs, which usually rely on a large amount of computing resources and time to construct and maintain scene graphs. These scene graphs need to describe every object and its position in the environment in detail so that the robot can navigate and operate between different scenes. However, this method has the disadvantage that when the scene changes, the scene graph needs to be updated, and the updating process is complex and time-consuming. In addition, these methods do not perform well in dynamic environments because the positions and states of objects in the environment may change frequently, causing the scene graph to fail. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provides a target-driven mobile grasping control method for a multi-scene environment. By combining rough scene priors and fine pose estimation, the method can achieve efficient and accurate multi-scene mobile grasping control, ensuring that the robot can accurately perform tasks in a complex multi-scene environment.

[0006] The purpose of the present application can be achieved by the following technical solution: a target-driven mobile grasping control method for a multi-scene environment, comprising the following steps:

[0007] S1, input the RGB-D sampling sequence of the multi-sub-scene, and initialize the 3D Gaussian model of the whole scene by using the sparse 3D point cloud with semantics, i.e. the coarse-grained Gaussian model;

[0008] S2, obtain the first depth information of the target object from the coarse-grained Gaussian model, input the pre-constructed pose estimation network model, and output the approximate pose of the target object;

[0009] Based on the approximate pose of the target object, the mobile chassis of the robot is controlled to move to approach the target object;

[0010] S3, during the movement of the robot, the diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model;

[0011] S4, obtain the second depth information of the target object from the fine-grained Gaussian model, input the pre-constructed pose estimation network model, and output the precise pose of the target object, i.e. the starting pose of the robot operation;

[0012] S5, Gaussian editing is performed on the fine-grained Gaussian model, and the depth information corresponding to the operation end of the edited object in the local scene is input into the pre-constructed pose estimation network model to obtain the operation end pose of the robot;

[0013] S6, according to the starting pose and the ending pose of the robot operation, the robot manipulator is controlled to complete the corresponding action.

[0014] Further, the step S1 is specifically for the RGB-D sampling sequence of the multi-sub-scene, and the coarse high-level expression of the whole scene is reconstructed based on the camera pose to obtain the 3D Gaussian model.

[0015] Further, the pose estimation network model in steps S2, S4 and S5 specifically adopts the MVPoseNet6D model architecture.

[0016] Further, the construction process of the pose estimation network model includes: constructing a simulation synthetic dataset, and training the simulation-to-real pose estimation network model based on the constructed data.

[0017] Further, the working process of the pose estimation network model includes:

[0018] First, input an RGB image and a depth image to be processed, i.e. an RGB-D sequence, and perform preprocessing operations;

[0019] Then, input the RGB-D sequence into the pre-trained Grounded-Light-HQSAM model to obtain the detection box and label of the target object;

[0020] Then, the features output by the Grounded-Light-HQ SAM model are input into the MVPoseNet6D model for 6-DoF (6-Degree of Freedom) target pose estimation;

[0021] Finally, the MVPoseNet6D model is trained using synthetic multi-view data to capture the pose of the target object from different perspectives and generate accurate target object pose and 3D size information.

[0022] Further, the step S3 specifically enhances the depth information by an implicit diffusion model to complete the coarse-grained Gaussian model to obtain a fine-grained Gaussian model.

[0023] Further, the implicit diffusion model specifically adopts a text-to-image diffusion model Stable Diffusion.

[0024] Further, the pre-training process of the implicit diffusion model includes:

[0025] First, a large-scale SceneFlow dataset is used, which contains rich image frames with depth information, and a mask is randomly generated at the beginning to simulate different missing areas;

[0026] Then, the pre-trained Stable Diffusion v1.5 model is used to initialize the diffusion model with the pre-trained depth prediction weights of the Marigold model;

[0027] Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using a VAE encoder to generate embedding vectors;

[0028] Then, the mask is down-sampled and combined with the encoded latent vectors to generate a feature map;

[0029] The combined feature map is then input into a U-Net-based denoiser to iteratively remove noise, and the U-Net parameters are updated according to the DDPM strategy to optimize the denoising diffusion objective function;

[0030] Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss value reaches the stopping requirement, and the training is completed.

[0031] Further, the step S3 specifically includes the following steps:

[0032] S31, input the depth map, corresponding color image and mask into the implicit diffusion model;

[0033] S32, copy the single-channel depth map into three channels to form an RGB-like input, and perform linear normalization processing;

[0034] S33, encode the normalized depth map and color image into the latent space using a pre-trained VAE encoder to generate corresponding embedding vectors;

[0035] S34, add noise to the encoded depth embedding vector to generate a noise embedding vector;

[0036] S35, downsample the mask mask of the target region to match the encoded embedding vector;

[0037] S36, connect the noise latent depth code, clean latent depth code and mask, and latent image code together to generate a combined feature map;

[0038] S37, input the combined feature map into a U-Net-based denoiser to recover the clean latent image through multiple steps of denoising;

[0039] S38, project the completed depth map and color image from the image space to the 3D coordinates, merge the point cloud and perform 3D Gaussian fine-tuning to obtain the optimized Gaussian model, i.e. the fine-grained Gaussian model.

[0040] Further, the step S5 is specifically using a Gaussian expression editing method to edit the fine-grained Gaussian model, so that the scene update is mapped to the Gaussian expression update to realize Gaussian repair.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] The present application uses a sparse 3D point cloud with semantics to initialize the 3D Gaussian model of the entire scene for the RGB-D sequence with camera pose; then the approximate pose of the target is obtained by querying the coarse-grained Gaussian, the mobile chassis is controlled to move to approach the target, and the range of pose estimation of the target is reduced to the corresponding sub-scene; and during the movement, the fine-grained Gaussian is optimized by using a diffusion-based depth completion to obtain the accurate pose of the target. Thus, the rough scene prior and the fine pose estimation are combined to obtain the accurate pose of the starting point and the end point in a "think first, then do" manner, and the related sub-scene is updated through 3D scene repair technology to ensure that the robot arm can accurately perform the task, and flexible navigation and fine operation are realized in complex and variable environments without the need for a large amount of scene dataset annotation and training, and the present application has good generalization ability and robustness.

[0043] The application is based on a scene coarse-grained Gaussian model with semantics and mobile navigation technology, and can efficiently obtain the preliminary pose of a target object in a multi-sub-scene environment and navigate a mobile chassis to approach the target object. This process significantly reduces the dependence on detailed modeling of a single scene, and improves the response speed and operation efficiency of the system.

[0044] The application adopts a diffusion-based depth completion technology to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model, and inputs the second depth information of the target object obtained from the fine-grained Gaussian model into a pose estimation model, so as to obtain the accurate pose of the target. The depth completion technology optimizes the depth map, and can still provide high-quality depth information even in the case of noise and occlusion, thereby effectively improving the accuracy of pose estimation.

[0045] The application adopts a Gaussian repair method, and performs Gaussian editing on the fine-grained Gaussian model through a Gaussian expression editing mode, so that the scene update is mapped to the Gaussian expression update. Even if the positions and states of objects in the environment change, the pose estimation and operation planning can still be updated in time, the efficient and accurate operation performance is maintained, and stable operation in a dynamically changing environment can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The figure is a method flowchart of the application;

[0047] Figure 2 The figure is an application architecture diagram built in the embodiment;

[0048] Figure 3 The figure is an application process diagram of the embodiment;

[0049] Figure 4 The figure is a diffusion-based depth completion process diagram in the embodiment;

[0050] Figure 5 The figure is a working process diagram of the target pose estimation model in the embodiment;

[0051] Figure 6 The figure is a process diagram of updating and editing the local scene in the embodiment. DETAILED DESCRIPTION

[0052] The application will be described in detail below in combination with the drawings and specific embodiments.

[0053] EMBODIMENT

[0054] The scheme takes into account that in recent years, with the development of deep learning and large model technology, large-scale deep neural networks are gradually being used to solve challenges in robot scene perception and operation. Deep neural network models, large models usually have a large number of parameters and complex network structures, and can learn rich features and patterns from massive data. Through fine-tuning on specific datasets, these models can adapt to different environments and scenarios, providing high-precision target detection and segmentation results, and performing well in scene semantic segmentation, scene understanding and other tasks. Using the above technology not only improves the perception and operation capabilities of robots in complex environments, but also significantly reduces the dependence on a large amount of labeled data, enabling robots to complete tasks more efficiently and accurately. Therefore, a target-driven mobile grasping control method for multi-scene environments is designed, as shown in Figure 1 The method comprises the following steps:

[0055] S1, input the RGB-D sampling sequence of multiple sub-scenes, and initialize the 3D Gaussian model of the entire scene using a sparse 3D point cloud with semantics, i.e. a coarse-grained Gaussian model;

[0056] S2, query the first depth information of the target object from the coarse-grained Gaussian model, input a pre-constructed pose estimation network model, and output the approximate pose of the target object;

[0057] Based on the approximate pose of the target object, the mobile chassis of the robot is controlled to move to approach the target object;

[0058] S3, during the movement of the robot, a diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model;

[0059] S4, query the second depth information of the target object from the fine-grained Gaussian model, input a pre-constructed pose estimation network model, and output the precise pose of the target object, i.e. the starting pose for robot operation;

[0060] S5, edit the fine-grained Gaussian model, input the depth information of the object operation endpoint after local scene editing into a pre-constructed pose estimation network model, and obtain the robot operation endpoint pose;

[0061] S6, according to the robot operation starting pose and the robot operation endpoint pose, control the robot manipulator to complete the corresponding action.

[0062] The embodiment applies the above scheme to fine operation of a desktop target object, builds an application framework as shown in Figure 2 The specific application process is as shown in Figure 3 The main content includes:

[0063] Step 1, input the RGB-D sequence with camera pose, initialize the 3D Gaussian model of the whole scene through sparse 3D point cloud;

[0064] Step 2, obtain the approximate pose of the target object by querying the coarse-grained Gaussian model, and navigate the mobile chassis to approach the target object;

[0065] Step 3, during the movement, the fine-grained Gaussian model is obtained by using the diffusion-based depth completion technology, and the starting accurate pose of the target object is estimated;

[0066] Step 4, the accurate pose of the operation starting point and the ending point is obtained in the "think first, then do" way by using the Gaussian repair method, and the fine operation under the target driving is completed.

[0067] In step 2, the object depth in the coarse-grained Gaussian scene is input into the pose estimation network model to obtain the approximate pose of the query object;

[0068] In step 3, the object depth in the optimized fine-grained Gaussian scene is input into the pose estimation network model to obtain the starting accurate pose of the query object;

[0069] In step 4, the depth information corresponding to the operation ending point of the object in the edited local scene is input into the pose estimation network model to obtain the ending accurate pose of the target object.

[0070] In this scheme, a set of rendering methods for class-level pose and size estimation dataset from simulation to reality are constructed, and a depth-based network model is trained. The pose estimation network model is trained based on pure simulation data rendering, which can obtain the cross-domain inference effect in the real world.

[0071] In this embodiment, the text-to-image diffusion model Stable Diffusion is selected as the depth completion network model, as shown in Figure 4 The depth completion process of the network model is as follows:

[0072] First, input the depth map, the corresponding color image and the mask into the depth completion model;

[0073] Then, copy the single-channel depth map into three channels to form an RGB-like input, and perform linear normalization processing;

[0074] Next, use the pre-trained VAE encoder to encode the normalized depth map and color image into the latent space to generate the corresponding embedding vector;

[0075] Next, the encoded depth embedding vector is subjected to noise processing to generate a noise embedding vector;

[0076] Next, the mask of the target region is down-sampled to match the encoded embedding vector;

[0077] Next, the noise latent depth code, the clean latent depth code and the mask, and the latent image code are connected together to generate a combined feature map;

[0078] Next, the combined feature map is input into a U-Net-based denoiser to recover the clean latent image through multi-step denoising;

[0079] Finally, the completed depth map and color image are back-projected from the image space to 3D coordinates, the point cloud is merged and 3D Gaussian fine-tuning is performed to obtain an optimized Gaussian model.

[0080] The pre-training step of the diffusion model is as follows:

[0081] First, a large-scale SceneFlow dataset is used, which contains rich image frames with depth information, and a mask is randomly generated at the beginning to simulate different missing regions;

[0082] Then, the pre-trained Stable Diffusion v1.5 model is used to initialize the diffusion model with the pre-trained depth prediction weights of the Marigold model;

[0083] Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using a VAE encoder to generate an embedding vector;

[0084] Next, the mask is down-sampled and combined with the encoded latent vector to generate a feature map;

[0085] Next, the combined feature map is input into a U-Net-based denoiser to iteratively remove noise and update the U-Net parameters according to the DDPM strategy to optimize the denoising diffusion objective function;

[0086] Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss value meets the stopping requirement, and the training is completed.

[0087] In this embodiment, the target pose estimation network model is MVPoseNet6D, as shown in Figure 5 The process of pose estimation using this network structure is as follows:

[0088] First, input an RGB image and a depth image (RGB-D sequence) to be processed and perform preprocessing operations;

[0089] Then, the RGB-D sequence is input into the pre-trained Grounded-Light-HQSAM model to obtain the bounding box and label of the target object;

[0090] Next, the features output by the Grounded-Light-HQSAM model are input into the MVPoseNet6D model to perform 6-DoF (6-DoF) target pose estimation;

[0091] Finally, the MVPoseNet6D model is trained using synthetic multi-view data, which can capture the pose of the target object from different perspectives and generate accurate target object pose and 3D size information.

[0092] Figure 6 The process shown is a three-dimensional editing after scene perception three-dimensional reconstruction, using local object segmentation and updating the edited scene by filling in the scene depth through the diffusion model.

[0093] Through the above process, the depth completion process can be effectively completed, and the accurate pose of the starting point and ending point of the operation is obtained, and high-precision target pose estimation and grasping operation are realized in a multi-scene environment.

[0094] As can be seen from the above, the scheme combines rough scene priori and fine pose estimation to obtain the accurate pose of the starting point and ending point of the operation in a "think first, then do" manner, updates the relevant sub-scene through 3D scene repair technology, ensures that the robot arm can accurately perform the task, realizes flexible navigation and fine operation in a complex and variable environment, and does not require a large amount of scene dataset annotation and training. The embodiment verification can effectively and efficiently realize accurate operation tasks in a multi-scene desktop environment, the model is trained and optimized in stages, avoiding large-scale retraining, and providing accurate and effective operation results.

Claims

1. A target-driven mobile grasping control method for multi-scene environments, characterized in that, The method comprises the following steps: S1, inputting an RGB-D sampling sequence of a multi-sub-scene, initializing a 3D Gaussian model of the whole scene by using a sparse 3D point cloud with semantics, that is, a coarse-grained Gaussian model, specifically, based on camera poses, reconstructing a rough high-level expression of the whole scene for the RGB-D sampling sequence of the multi-sub-scene to obtain a 3D Gaussian model; S2, obtaining first depth information of a target object from the coarse-grained Gaussian model, inputting a pre-constructed pose estimation network model, and outputting an approximate pose of the target object; Based on the approximate pose of the target object, the moving chassis of the robot is controlled to move to approach the target object; The construction process of the pose estimation network model comprises: constructing a simulation synthesis data set, and training a simulation-to-real pose estimation network model based on the constructed data; The working process of the pose estimation network model comprises: First, inputting an RGB image and a depth image to be processed, that is, an RGB-D sequence, and performing a preprocessing operation; Then, inputting the RGB-D sequence into a pre-trained Grounded-Light-HQSAM model to obtain a detection box and a label of the target object; Next, inputting the features output by the Grounded-Light-HQSAM model into an MVPoseNet6D model to perform 6-degree-of-freedom target pose estimation; Finally, the MVPoseNet6D model is trained by using multi-view synthetic data to capture the pose of the target object from different perspectives and generate accurate target object pose and 3D size information; S3, during the movement of the robot, a diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model, specifically, the depth information is enhanced by an implicit diffusion model to complete the Gaussian expression of the coarse-grained Gaussian model to obtain the fine-grained Gaussian model; S4, obtaining second depth information of the target object from the fine-grained Gaussian model, inputting a pre-constructed pose estimation network model, and outputting an accurate pose of the target object, that is, a robot operation starting point pose; S5, performing Gaussian editing on the fine-grained Gaussian model, inputting the depth information corresponding to the object operation end point after local scene editing into a pre-constructed pose estimation network model to obtain a robot operation end point pose; S6, controlling the robot manipulator to complete corresponding actions according to the robot operation starting point pose and the robot operation end point pose.

2. The target-driven mobile grasping control method for multi-scene environments according to claim 1, wherein, The pose estimation network model in steps S2, S4 and S5 specifically adopts an MVPoseNet6D model architecture.

3. The target-driven mobile grasping control method for multi-scene environments according to claim 1, wherein, The implicit diffusion model specifically adopts a text-to-image diffusion model Stable Diffusion.

4. The target-driven mobile grasping control method for multi-scene environments according to claim 3, wherein, The pre-training process of the implicit diffusion model comprises: First, a large-scale SceneFlow data set is used, which contains rich image frames with depth information, and a mask is randomly generated at the beginning to simulate different missing areas; Then, using the pre-trained Stable Diffusion v1.5 model, the pre-trained deep prediction weights of the Marigold model are used to initialize the diffusion model; Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using the VAE encoder to generate embedding vectors; Then, the mask is down-sampled and combined with the encoded latent vectors to generate a combined feature map; The combined feature map is then input into the U-Net-based denoiser to iteratively remove noise, and the U-Net parameters are updated according to the DDPM strategy to optimize the denoising diffusion objective function. Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss function loss value meets the stopping requirement, and the training is completed.

5. The target-driven mobile grasping control method for multi-scene environments according to claim 1, wherein, The step S5 specifically adopts a Gaussian expression editing method to edit the fine-grained Gaussian model, so that the scene update is mapped to the Gaussian expression update to realize Gaussian repair.

Images