Target driving movement grabbing control method used in multi-scene environment

Abstract

The invention relates to a target driving movement grabbing control method used in a multi-scene environment, and the method comprises the steps: inputting an RGB-D sampling sequence of multiple sub-scenes, and initializing a 3D Gaussian model of the whole scene through employing a sparse 3D point cloud with semantics; acquiring an approximate pose of the target by querying coarse-grained Gaussian, navigating the mobile chassis to approach the target, and narrowing a pose estimation range of the target to a corresponding sub-scene; in the moving process, fine-grained gauss are optimized through diffusion-based depth completion, and the accurate pose of the target is obtained; gaussian editing is conducted on the fine-grained Gaussian model, and the robot operation end point pose is obtained through Gaussian repairing; and according to the operation starting point pose and the operation end point pose of the robot, a robot mechanical arm is controlled to complete corresponding actions. Compared with the prior art, the method has the advantages that rough scene prior and fine pose estimation are combined, efficient and accurate multi-scene moving grabbing control can be achieved, and it is ensured that the robot can accurately execute tasks in the complex multi-scene environment.

Description

Technical Field

[0001] The present invention relates to the technical field of robot operation control, and in particular to a target driven mobile grasping control method for use in a multi-scenario environment. Background Art

[0002] Service robots are mainly composed of a mobile chassis and a robotic arm. In daily tasks, service robots usually need to navigate and perform delicate operations in multiple sub-scene environments. However, most existing technologies focus on complex tasks in a single scene or simple tasks across scenes, and it is difficult to effectively handle complex tasks in multiple sub-scenes. They face the challenge of efficiently representing and coordinating the collaboration between the mobile chassis and the robotic arm in multiple sub-scenes.

[0003] Traditional methods rely on single-view observation or a large amount of computing resources to complete tasks, which is inefficient and inaccurate when dealing with cross-scene tasks. Specifically, traditional robotic systems rely mainly on complete modeling and planning of a single scene when dealing with complex tasks. These systems can complete tasks such as grasping, carrying, and stacking by finely identifying and manipulating objects in a single scene. However, when it is necessary to move between multiple different scenes and perform a series of complex operations, existing methods often seem to be unable to cope with the task and cannot obtain the precise position of the starting and ending points of the robot's operation.

[0004] Some studies have attempted to implement cross-scenario tasks by constructing 3D scene graphs, which usually rely on a lot of computing resources and time to build and maintain scene graphs. These scene graphs need to describe every object and its position in the environment in detail so that the robot can navigate and operate between different scenes. However, the disadvantage of this method is that when the scene changes, the scene graph needs to be updated again, and the update process is complicated and time-consuming. In addition, these methods do not perform well when dealing with dynamic environments, because the position and state of objects in the environment may change frequently, causing the scene graph to fail. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a target-driven mobile grasping control method for a multi-scene environment. By combining rough scene priors and fine pose estimation, efficient and accurate multi-scene mobile grasping control can be achieved, ensuring that the robot can accurately perform tasks in complex multi-scene environments.

[0006] The object of the present invention can be achieved by the following technical solution: A target driven mobile grasping control method for a multi-scenario environment comprises the following steps:

[0007] S1, input the RGB-D sampling sequence of multiple sub-scenes, and use the sparse 3D point cloud with semantics to initialize the 3D Gaussian model of the entire scene, which is a coarse-grained Gaussian model;

[0008] S2, querying and obtaining the first depth information of the target object from the coarse-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the approximate pose of the target object;

[0009] Based on the approximate position and posture of the target object, the mobile chassis of the robot is controlled to move to approach the target object;

[0010] S3. During the robot movement, the diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model;

[0011] S4, querying and obtaining the second depth information of the target object from the fine-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the precise pose of the target object, which is the starting pose of the robot operation;

[0012] S5. Perform Gaussian editing on the fine-grained Gaussian model, input the depth information corresponding to the object operation endpoint after local scene editing into the pre-built pose estimation network model, and obtain the robot operation endpoint pose;

[0013] S6. According to the robot operation starting position and the robot operation ending position, the robot arm is controlled to complete the corresponding action.

[0014] Furthermore, the step S1 specifically reconstructs a rough high-level expression of the entire scene based on the camera pose for the RGB-D sampling sequence of multiple sub-scenes to obtain a 3D Gaussian model.

[0015] Furthermore, the pose estimation network model in step S2, step S4 and step S5 specifically adopts the MVPoseNet6D model architecture.

[0016] Furthermore, the construction process of the pose estimation network model includes: constructing a simulated synthetic data set, and performing simulation to real pose estimation network model training based on the constructed data.

[0017] Furthermore, the working process of the pose estimation network model includes:

[0018] First, an RGB image and a depth image to be processed, i.e., an RGB-D sequence, are input and preprocessed;

[0019] Then, the RGB-D sequence is input into the pre-trained Grounded-Light-HQSAM model to obtain the detection box and label of the target object;

[0020] Next, the features output by the Grounded-Light-HQSAM model are input into the MVPoseNet6D model to perform 6-DoF target pose estimation.

[0021] Finally, the MVPoseNet6D model is trained using multi-view synthetic data to capture the pose of the target object from different perspectives and generate accurate pose and 3D size information of the target object.

[0022] Furthermore, the step S3 specifically enhances the depth information through an implicit diffusion model to complete the coarse-grained Gaussian model with Gaussian expression to obtain a fine-grained Gaussian model.

[0023] Furthermore, the implicit diffusion model specifically adopts a text-to-image diffusion model Stable Diffusion.

[0024] Furthermore, the pre-training process of the implicit diffusion model includes:

[0025] First, we use a large-scale SceneFlow dataset that contains rich image frames with depth information. We initially randomly generate masks to simulate different missing areas.

[0026] Then, use the pre-trained Stable Diffusion v1.5 model to initialize the diffusion model with the pre-trained deep prediction weights of the Marigold model;

[0027] Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using a VAE encoder to generate an embedding vector.

[0028] The mask is then downsampled and combined with the encoded latent vector to generate a feature map;

[0029] Then the combined feature map is input into the U-Net-based denoiser to iteratively remove noise, and the denoising diffusion objective function is optimized according to the DDPM strategy to update the U-Net parameters;

[0030] Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss function value reaches the stopping requirement, and the training is completed.

[0031] Furthermore, the step S3 specifically includes the following steps:

[0032] S31, inputting the depth map, the corresponding color image and the mask into the implicit diffusion model;

[0033] S32, copying the single-channel depth map into three channels to form an input similar to RGB, and performing linear normalization processing;

[0034] S33, using the pre-trained VAE encoder to encode the normalized depth map and color image into the latent space and generate the corresponding embedding vector;

[0035] S34, performing noise processing on the encoded depth embedding vector to generate a noise embedding vector;

[0036] S35, downsampling the mask of the target area to match the encoded embedding vector;

[0037] S36, connecting the noisy latent depth code, the clean latent depth code, the mask, and the latent image code together to generate a combined feature map;

[0038] S37, input the combined feature map into a U-Net based denoiser, and restore a clean latent map through multi-step denoising;

[0039] S38, back-projecting the completed depth map and color image from the image space to 3D coordinates, merging the point cloud and fine-tuning the 3D Gaussian to obtain an optimized Gaussian model, which is a fine-grained Gaussian model.

[0040] Furthermore, the step S5 specifically adopts a Gaussian expression editing method to perform Gaussian editing on the fine-grained Gaussian model, so that the scene update is mapped to the Gaussian expression update to achieve Gaussian repair.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention uses a sparse 3D point cloud with semantics to initialize the 3D Gaussian model of the entire scene for RGB-D sequences with camera poses; then obtains the approximate pose of the target by querying the coarse-grained Gaussian, controls the mobile chassis to move close to the target, and narrows the range of pose estimation for the target to the corresponding sub-scene; and during the movement, uses diffusion-based depth completion to optimize the fine-grained Gaussian to obtain the precise pose of the target. In this way, the rough scene prior and the fine pose estimation are combined to obtain the precise pose of the operation start and end point in a "think before doing" manner, and the relevant sub-scenes are updated through 3D scene repair technology to ensure that the robot's mechanical arm can accurately perform tasks, and realize flexible navigation and fine operation in complex and changeable environments. It does not require the annotation and training of a large number of scene data sets, and has good generalization ability and robustness.

[0043] Based on the semantic scene coarse-grained Gaussian model and mobile navigation technology, the present invention can efficiently obtain the preliminary position and pose of the target object in a multi-subscene environment and navigate the mobile chassis to approach the target object. This process significantly reduces the dependence on detailed modeling of a single scene and improves the system's response speed and operating efficiency.

[0044] The present invention adopts a diffusion-based depth completion technology to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model, and inputs the second depth information of the target object obtained from the fine-grained Gaussian model into the pose estimation model, so as to obtain the precise pose of the target. Among them, the depth completion technology optimizes the depth map and can provide high-quality depth information even in the presence of noise and occlusion, thereby effectively improving the accuracy of pose estimation.

[0045] The present invention adopts a Gaussian repair method and performs Gaussian editing on a fine-grained Gaussian model through Gaussian expression editing, so that scene updates are mapped to Gaussian expression updates. Even if the position and state of objects in the environment change, the pose estimation and operation planning can be updated in time, maintaining efficient and accurate operation performance, and being able to operate stably in a dynamically changing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 A schematic diagram of the application architecture constructed in the embodiment;

[0048] Figure 3 It is a schematic diagram of the application process of the embodiment;

[0049] Figure 4 Schematic diagram of a diffusion-based depth completion process in an embodiment;

[0050] Figure 5 Schematic diagram of the working process of the target pose estimation model in the embodiment;

[0051] Figure 6 It is a schematic diagram of the process of updating the edited local scene in the embodiment. DETAILED DESCRIPTION

[0052] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example

[0054] This scheme takes into account that in recent years, with the development of deep learning and large model technology, large-scale deep neural networks are gradually being used to solve the challenges in robot scene perception and operation. Deep neural network models and large models usually have a large number of parameters and complex network structures, and can learn rich features and patterns from massive data. By fine-tuning on specific data sets, these models can adapt to different environments and scenes, provide high-precision target detection and segmentation results, and perform well in tasks such as scene semantic segmentation and scene understanding. The use of the above technologies can not only improve the robot's perception and operation capabilities in complex environments, but also significantly reduce dependence on large amounts of labeled data, allowing robots to complete tasks more efficiently and accurately. Therefore, a target-driven mobile grasping control method for multi-scene environments is designed, such as Figure 1 As shown, the following steps are included:

[0055] S1, input the RGB-D sampling sequence of multiple sub-scenes, and use the sparse 3D point cloud with semantics to initialize the 3D Gaussian model of the entire scene, which is a coarse-grained Gaussian model;

[0056] S2, querying and obtaining the first depth information of the target object from the coarse-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the approximate pose of the target object;

[0057] Based on the approximate position and posture of the target object, the mobile chassis of the robot is controlled to move to approach the target object;

[0058] S3. During the robot movement, the diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model;

[0059] S4, querying and obtaining the second depth information of the target object from the fine-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the precise pose of the target object, which is the starting pose of the robot operation;

[0060] S5. Perform Gaussian editing on the fine-grained Gaussian model, input the depth information corresponding to the object operation endpoint after local scene editing into the pre-built pose estimation network model, and obtain the robot operation endpoint pose;

[0061] S6. According to the robot operation starting position and the robot operation ending position, the robot arm is controlled to complete the corresponding action.

[0062] This embodiment applies the above solution to perform fine operations on desktop target objects, and builds Figure 2 The application framework shown in the figure is as follows: Figure 3 As shown, the main contents are:

[0063] Step 1: Input the RGB-D sequence with camera pose and initialize the 3D Gaussian model of the entire scene through the sparse 3D point cloud;

[0064] Step 2: Obtain the approximate pose of the target object by querying the coarse-grained Gaussian model, and navigate the mobile chassis to approach the target object;

[0065] Step 3: During the movement, a diffusion-based depth completion technique is used to optimize the fine-grained Gaussian model and estimate the precise position of the starting point of the target object;

[0066] Step 4: Use the Gaussian repair method to obtain the precise position of the operation start and end points in a "think first, do later" manner to complete the target-driven fine operation.

[0067] Among them, in step 2, specifically, the depth of the object queried in the coarse-grained Gaussian scene is input into the pose estimation network model to obtain the approximate pose of the queried object;

[0068] In step 3, the depth of the object queried in the optimized fine-grained Gaussian scene is input into the pose estimation network model to obtain the precise pose of the starting point of the queried object;

[0069] In step 4, the depth information corresponding to the end point of the object operation after the local scene editing is input into the pose estimation network model to obtain the precise end point pose of the target object.

[0070] In this solution, we constructed a set of rendering methods for class-level pose and size estimation datasets from simulation to reality, as well as depth-based network model training. The pose estimation network model was trained based on pure simulation data rendering, which can achieve cross-domain reasoning effects in the real world.

[0071] In this embodiment, the text-to-image diffusion model Stable Diffusion is selected as the deep completion network model, such as Figure 4 As shown in the figure, the process of depth completion of the network model is as follows:

[0072] First, the depth map, the corresponding color image and the mask are input to the depth completion model;

[0073] Then, the single-channel depth map is copied into three channels to form an RGB-like input and linearly normalized;

[0074] Next, the normalized depth map and color image are encoded into the latent space using the pre-trained VAE encoder to generate the corresponding embedding vectors.

[0075] Next, the encoded deep embedding vector is subjected to noise processing to generate a noise embedding vector;

[0076] Next, the mask of the target area is downsampled to match the encoded embedding vector;

[0077] Next, the noisy latent depth code, clean latent depth code, mask, and latent image code are concatenated together to generate a combined feature map;

[0078] Next, the combined feature map is input into a U-Net-based denoiser to restore a clean latent map through multi-step denoising;

[0079] Finally, the completed depth map and color image are back-projected from the image space to 3D coordinates, the point cloud is merged and the 3D Gaussian is fine-tuned to obtain the optimized Gaussian model.

[0080] The pre-training steps of the diffusion model are:

[0081] First, we use a large-scale SceneFlow dataset that contains rich image frames with depth information. We initially randomly generate masks to simulate different missing areas.

[0082] Then, use the pre-trained Stable Diffusion v1.5 model to initialize the diffusion model with the pre-trained deep prediction weights of the Marigold model;

[0083] Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using a VAE encoder to generate an embedding vector.

[0084] Next, the mask is downsampled and combined with the encoded latent vector to generate a feature map;

[0085] Next, the combined feature map is input into the U-Net-based denoiser to iteratively remove noise, and the denoising diffusion objective function is optimized according to the DDPM strategy to update the U-Net parameters;

[0086] Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss value reaches the stopping requirement, and the training is completed.

[0087] In this embodiment, the target pose estimation network model uses MVPoseNet6D, such as Figure 5 As shown, the process of pose estimation of the network structure is:

[0088] First, input an RGB image and depth image (RGB-D sequence) to be processed and perform preprocessing operations;

[0089] Then, the RGB-D sequence is input into the pre-trained Grounded-Light-HQSAM model to obtain the detection box and label of the target object;

[0090] Next, the features output by the Grounded-Light-HQSAM model are input into the MVPoseNet6D model to perform 6-DoF target pose estimation.

[0091] Finally, the MVPoseNet6D model is trained using multi-view synthetic data, which can capture the pose of the target object from different perspectives and generate accurate pose and 3D size information of the target object.

[0092] Figure 6 The process shown is for 3D editing after scene-aware 3D reconstruction, using local object segmentation and completing the scene depth through a diffusion model to update the edited scene.

[0093] Through the above process, the depth completion process can be effectively completed, thereby obtaining the precise pose of the starting and ending points of the operation, and realizing high-precision target pose estimation and grasping operations in multi-scene environments.

[0094] In summary, this solution combines rough scene priors with fine pose estimation to obtain the precise poses of the operation start and end points in a "think before you act" manner, and updates relevant sub-scenes through 3D scene repair technology to ensure that the robot arm can accurately perform tasks, achieving flexible navigation and fine operations in complex and changing environments without the need for annotation and training of large amounts of scene data sets. The implementation examples have proven that precise operation tasks can be effectively and efficiently achieved in a multi-scene desktop environment, and large-scale retraining can be avoided by training and optimizing the model in stages, while providing accurate and effective operation results.

Claims

1. A target driven mobile grasping control method for a multi-scenario environment, characterized in that: The following steps are involved: S1, input the RGB-D sampling sequence of multiple sub-scenes, and use the sparse 3D point cloud with semantics to initialize the 3D Gaussian model of the entire scene, which is a coarse-grained Gaussian model; S2, querying and obtaining the first depth information of the target object from the coarse-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the approximate pose of the target object; Based on the approximate position and posture of the target object, the mobile chassis of the robot is controlled to move to approach the target object; S3. During the robot movement, the diffusion-based depth completion technology is used to optimize the coarse-grained Gaussian model to obtain a fine-grained Gaussian model; S4, querying and obtaining the second depth information of the target object from the fine-grained Gaussian model, inputting the pre-built pose estimation network model, and outputting the precise pose of the target object, which is the starting pose of the robot operation; S5. Perform Gaussian editing on the fine-grained Gaussian model, input the depth information corresponding to the object operation endpoint after the local scene editing into the pre-built pose estimation network model, and obtain the robot operation endpoint pose; S6. According to the robot operation starting position and the robot operation ending position, the robot arm is controlled to complete the corresponding action.

2. The target driven mobile grasping control method for multi-scenario environment according to claim 1, characterized in that: The step S1 specifically reconstructs a rough high-level expression of the entire scene based on the camera pose for the RGB-D sampling sequence of multiple sub-scenes to obtain a 3D Gaussian model.

3. The target driven mobile grasping control method for multi-scenario environment according to claim 1, characterized in that: The pose estimation network model in step S2, step S4 and step S5 specifically adopts the MVPoseNet6D model architecture.

4. The target driven mobile grasping control method for multi-scenario environment according to claim 3, characterized in that: The construction process of the pose estimation network model includes: constructing a simulation synthetic data set, and performing simulation to real pose estimation network model training based on the constructed data.

5. The target driven mobile grasping control method for multi-scenario environment according to claim 4, characterized in that: The working process of the pose estimation network model includes: First, an RGB image and a depth image to be processed, i.e., an RGB-D sequence, are input and preprocessed; Then, the RGB-D sequence is input into the pre-trained Grounded-Light-HQSAM model to obtain the detection box and label of the target object; Next, the features output by the Grounded-Light-HQSAM model are input into the MVPoseNet6D model to perform 6-DOF target pose estimation. Finally, the MVPoseNet6D model is trained using multi-view synthetic data to capture the pose of the target object from different perspectives and generate accurate pose and 3D size information of the target object.

6. The target driven mobile grasping control method for multi-scenario environment according to claim 1, characterized in that: The step S3 specifically enhances the depth information through an implicit diffusion model to complete the coarse-grained Gaussian model with Gaussian expression to obtain a fine-grained Gaussian model.

7. The target driven mobile grasping control method for multi-scenario environment according to claim 6, characterized in that: The implicit diffusion model specifically adopts the text-to-image diffusion model Stable Diffusion.

8. The target driven mobile grasping control method for multi-scenario environment according to claim 7, characterized in that: The pre-training process of the implicit diffusion model includes: First, we use a large-scale SceneFlow dataset that contains rich image frames with depth information. We initially randomly generate masks to simulate different missing areas. Then, use the pre-trained Stable Diffusion v1.5 model to initialize the diffusion model with the pre-trained deep prediction weights of the Marigold model; Next, the depth map is linearly normalized, and the normalized depth map and color image are encoded into the latent space using a VAE encoder to generate an embedding vector. The mask is then downsampled and combined with the encoded latent vector to generate a feature map; Then the combined feature map is input into the U-Net-based denoiser to iteratively remove noise, and the denoising diffusion objective function is optimized according to the DDPM strategy to update the U-Net parameters; Finally, the training data is input into the model with updated parameters, and the above steps are repeated until the loss function value reaches the stopping requirement, and the training is completed.

9. The target driven mobile grasping control method for multi-scenario environment according to claim 8, characterized in that: The step S3 specifically comprises the following steps: S31, inputting the depth map, the corresponding color image and the mask into the implicit diffusion model; S32, copying the single-channel depth map into three channels to form an input similar to RGB, and performing linear normalization processing; S33, using the pre-trained VAE encoder to encode the normalized depth map and color image into the latent space and generate the corresponding embedding vector; S34, performing noise processing on the encoded depth embedding vector to generate a noise embedding vector; S35, downsampling the mask of the target area to match the encoded embedding vector; S36, connecting the noisy latent depth code, the clean latent depth code, the mask, and the latent image code together to generate a combined feature map; S37, input the combined feature map into a U-Net based denoiser, and restore a clean latent map through multi-step denoising; S38, back-projecting the completed depth map and color image from the image space to 3D coordinates, merging the point cloud and fine-tuning the 3D Gaussian to obtain an optimized Gaussian model, which is a fine-grained Gaussian model.

10. The target driven mobile grasping control method for multi-scenario environment according to claim 1, characterized in that: The step S5 specifically adopts a Gaussian expression editing method to perform Gaussian editing on the fine-grained Gaussian model, so that the scene update is mapped to the Gaussian expression update to achieve Gaussian repair.

Images