Method and storage medium for acquiring 6D pose of class-level target based on language description

By constructing a two-dimensional positioning model of language features and image features, combining three-dimensional point cloud segmentation and pose estimation network, the problem of positioning unknown objects in three-dimensional space is solved, and the high accuracy and practicality of six-degree-of-free pose pose acquisition is achieved. , suitable for robot crawling and human-computer interaction.

CN114399515BActive Publication Date: 2024-11-26FUDAN UNIVERSITY
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202111598303.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-24
Publication Date
2024-11-26
Estimated Expiration
2041-12-24

AI Technical Summary

Technical Problem

The prior art is difficult to accurately locate unknown objects and target objects in scenes in three-dimensional space, especially when robots interact with external objects, and lack effective three-dimensional visual positioning capabilities.

Method used

By constructing a two-dimensional object positioning model based on language features and image features, generating a two-dimensional bounding box, back-projecting to obtain point cloud data, using a three-dimensional convolutional network for point cloud segmentation, combining a class-level pose estimation network, estimating the target object's Six degrees of freedom posture.

Benefits of technology

It realizes high accuracy and practical target object positioning in three-dimensional space, can effectively remove background information, improves the accuracy of six-degree-of-freedom pose estimation, and supports voice-controlled robot grabbing and human-computer interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114399515B_ABST
    Figure CN114399515B_ABST
Patent Text Reader

Abstract

The present invention relates to a class-level target object 6D posture acquisition method based on language description and a storage medium, wherein the 6D posture acquisition method comprises: constructing a two-dimensional positioning model of an object based on language features and image features; inputting a picture and a language description into the two-dimensional positioning model, thereby generating a two-dimensional positioning of a target object in the picture according to the language description; back-projecting the two-dimensional positioning to the depth image using camera internal parameters to obtain point cloud data corresponding to the target area; constructing a point cloud segmentation model for removing background point clouds, inputting the target area point cloud into the point cloud segmentation model to obtain point cloud data of the target object; constructing a class-level six-degree-of-freedom posture estimation model for estimating the posture of an object in three-dimensional space, and inputting the target object point cloud data into the class-level object posture estimation model to obtain the posture of the target object. Compared with the prior art, the present invention has the advantages of high accuracy and strong practicality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of visual positioning technology, and in particular to a method and storage medium for acquiring 6D pose of a class-level target based on language description. Background Art

[0002] Visual localization is an important issue that connects visual information and language information. With the improvement of deep neural network modeling capabilities, visual localization can detect the target position that conforms to the language description in the image space through the fusion and utilization of multimodal information. However, due to the complex relationships between image descriptions, understanding the relationships and attributes between multiple objects in the language description is a current challenge. At the same time, the existing methods mainly focus on positioning in the image plane space, but in fields such as machine grasping and human-computer interaction, inferring the position and posture of objects in three-dimensional space is the basis for robots to interact with external objects.

[0003] In recent years, research related to visual localization has been committed to solving the complex relational structure in image descriptions, but related work is limited to positioning in two-dimensional plane space. Although the current mainstream visual localization methods are committed to learning the dependency between visual and language features to accurately locate the position of the target object in the image plane space, they lack the ability to be applied in three-dimensional scenes.

[0004] In recent years, the goal of visual positioning methods in three-dimensional scenes is to solve the task of visual positioning of objects in three-dimensional indoor space. Currently, three-dimensional visual positioning is limited to the positioning of known instances in indoor scenes, but there is still room for exploration of three-dimensional visual positioning of unknown objects and scenes. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and storage medium for acquiring 6D pose of class-level targets based on language description with high accuracy and strong practicality.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for acquiring 6D pose of a class-level target object based on language description, the method comprising:

[0008] Step 1: Construct a two-dimensional object positioning model based on language features and image features;

[0009] Step 2: Input the image and language description into the two-dimensional positioning model, so as to generate the two-dimensional positioning of the target object in the image according to the language description;

[0010] Step 3: Back-project the two-dimensional positioning depth image using the camera internal parameters to obtain the point cloud data corresponding to the target area;

[0011] Step 4: Build a point cloud segmentation model for removing background point clouds, and input the target area point cloud into the point cloud segmentation model to obtain point cloud data of the target object;

[0012] Step 5: Construct a class-level six-degree-of-freedom pose estimation model for estimating the pose of an object in three-dimensional space, and input the target object point cloud data into the class-level object pose estimation model to obtain the pose of the target object.

[0013] Preferably, the step 1 is specifically:

[0014] Step 1-1: Construct a language-image coupling confidence module;

[0015] Step 1-2: Calculate the confidence of the fit between the input text and each candidate bounding box image.

[0016] More preferably, the language-image coupling confidence module includes:

[0017] The object subject fit confidence unit fuses the image feature embedding code and the language feature embedding code and sends them to the encoder to regress and generate the fit confidence of the target subject described in the text and the object subject in the picture;

[0018] The object position fit confidence unit uses the real annotated bounding box information to generate the position embedding code, and predicts the fit confidence of the target position described in the text and the object position in the image through the encoder with the language feature embedding code;

[0019] The object proximity relationship fit confidence unit uses language feature embedding coding and position embedding coding of neighboring candidate regions to predict the fit confidence of the target-surrounding object relationship described in the text and the object-surrounding object relationship in the image through the encoder.

[0020] More preferably, the step 1-1 is specifically as follows:

[0021] The input image is embedded and encoded by extracting image features through the residual neural network ResNet, and the language description is embedded and encoded by extracting language features through the bidirectional recurrent neural network Bi-RNN. Then, the object subject fit confidence, object position fit confidence and object proximity relationship fit confidence are calculated respectively.

[0022] More preferably, the encoder is a multi-layer perceptron.

[0023] More preferably, the steps 1-2 are specifically as follows:

[0024] The input image is passed through the region proposal network (RPN) to generate multiple candidate bounding boxes. The image corresponding to each bounding box and the position information of its neighboring bounding boxes are taken as input. The weighted sum of the three confidences obtained in step 1-1 is taken as the sum of the confidences of the bounding box. The candidate bounding box with the highest confidence will be used as the target predicted bounding box, and then the image corresponding to the target predicted bounding box will be sent to Mask-RCNN for further prediction to obtain the category segmentation mask of the object.

[0025] Preferably, the step 3 is specifically:

[0026] After obtaining the object bounding box or segmentation mask, the corresponding depth image is intercepted accordingly, and this part of the depth image is back-projected through the camera intrinsic parameters to obtain the corresponding three-dimensional point cloud data in the area.

[0027] Preferably, the step 4 is specifically:

[0028] A point cloud segmentation model is built based on the three-dimensional convolutional network 3D-GCN to segment the object point cloud and the background point cloud, output the category label of each point cloud for distinction, and then delete the background point cloud.

[0029] Preferably, the step 5 is specifically:

[0030] The class-level pose estimation network DONet is used to build a class-level six-degree-of-freedom pose estimation model. The category and point cloud data of the object are input. The class-level six-degree-of-freedom pose estimation model extracts the geometric features of the object and estimates the rotation and translation of the target object in the camera coordinate system to obtain the six-degree-of-freedom pose of the target object.

[0031] A storage medium stores the method for acquiring 6D pose of a class-level target object described in any one of the above items.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. High accuracy: The class-level target 6D pose acquisition method in the present invention proposes the task of three-dimensional positioning of objects in the scene through language guidance in a machine grasping scenario, that is, it is composed of a two-stage network of two-dimensional visual positioning and six-degree-of-freedom pose estimation: using the two-dimensional visual positioning module to perform bounding box positioning and segmentation mask of the object in the image plane according to the language description; using the six-degree-of-freedom pose estimation network module, given the object bounding box or mask, the six-degree-of-freedom pose of a single or multiple objects is estimated; finally, the two-dimensional visual positioning module and the six-degree-of-freedom pose estimation module are combined to locate the specific target in the scene using the language description; in addition, the three-dimensional segmentation network provides the 6D pose acquisition method in the present invention with robustness to outliers in the point cloud data captured by the real depth sensor, that is, effectively removes background information and improves the accuracy of the six-degree-of-freedom pose estimation of the object.

[0034] 2. Strong practicability: The class-level target 6D pose acquisition method of the present invention can be applied to the industrial field and the field of household robots. It provides a method for directly controlling the robot to grasp through voice, realizes the human-computer interaction system from voice to control, has a high generalization ability for various types of objects, and is highly practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the process of acquiring 6D pose of class-level target in the present invention;

[0036] Figure 2 Schematic diagram of the structure of a language-image consistency confidence calculation module in an embodiment of the present invention;

[0037] Figure 3 It is a structural schematic diagram of a method for class-level six-degree-of-freedom pose estimation of a target object based on language description in an embodiment of the present invention;

[0038] Figure 4 It is a task diagram of a method for class-level six-degree-of-freedom pose estimation of a target object based on language description in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] A method for acquiring 6D pose of class-level objects based on language description, the process is as follows Figure 1 and Figure 3 As shown, including:

[0041] Step 1: Construct a two-dimensional object positioning model based on language features and image features;

[0042] Step 1-1: Construct a language-image coupling confidence module, specifically:

[0043] The input image is embedded and coded by extracting image features through the residual neural network ResNet, and the language description is embedded and coded by extracting language features through the bidirectional recurrent neural network Bi-RNN. Then, the object subject fit confidence, object position fit confidence, and object neighbor relationship fit confidence are calculated respectively.

[0044] like Figure 2 As shown, the language-image coupling confidence module includes:

[0045] The object subject fit confidence unit fuses the image feature embedding code and the language feature embedding code and sends them to the encoder to regress and generate the fit confidence of the target subject described in the text and the object subject in the picture;

[0046] The object position fit confidence unit uses the real annotated bounding box information to generate the position embedding code, and predicts the fit confidence of the target position described in the text and the object position in the image through the encoder with the language feature embedding code;

[0047] The object proximity relationship fit confidence unit uses language feature embedding coding and position embedding coding of neighboring candidate regions to predict the fit confidence of the target-surrounding object relationship described in the text and the object-surrounding object relationship in the image through the encoder;

[0048] The encoder in this embodiment adopts a multi-layer perceptron.

[0049] Step 1-2: Calculate the confidence level of the fit between the input text and each candidate bounding box image, specifically:

[0050] The input image is passed through the region proposal network RPN to generate multiple candidate bounding boxes. The image corresponding to each bounding box and the position information of its neighboring bounding boxes are used as input. The weighted sum of the three confidences obtained in step 1-1 is taken as the sum of the confidences of the bounding box. The candidate bounding box with the highest confidence will be used as the target predicted bounding box, and then the image corresponding to the target predicted bounding box is sent to Mask-RCNN for further prediction to obtain the category segmentation mask of the object.

[0051] Step 2: Input the image and language description into the two-dimensional positioning model, so as to generate the two-dimensional positioning of the target object in the image according to the language description, specifically:

[0052] Convert the speech description into text description, input the text description and the scene image into the object 2D localization model to generate the 2D localization of the target object, which is represented by a bounding box or a segmentation mask;

[0053] Step 3: Back-project the two-dimensional positioning depth image using the camera internal parameters to obtain the point cloud data corresponding to the target area, specifically:

[0054] The scene image is an RGB-D (color-depth) aligned image. Therefore, after obtaining the object bounding box or segmentation mask, the corresponding depth image is intercepted accordingly, and this part of the depth image is back-projected through the camera intrinsic parameters to obtain the corresponding 3D point cloud data in the area;

[0055] Step 4: Build a point cloud segmentation model for removing background point clouds, and input the target area point cloud into the point cloud segmentation model to obtain the point cloud data of the target object. Specifically:

[0056] A point cloud segmentation model is built based on a three-dimensional convolutional network 3D-GCN to output the category label of each point cloud for distinction, segment the object point cloud and the background point cloud, and then delete the background point cloud;

[0057] Step 5: Construct a class-level six-degree-of-freedom pose estimation model for estimating the pose of an object in three-dimensional space, and input the target object point cloud data into the class-level object pose estimation model to obtain the pose of the target object, specifically:

[0058] The class-level pose estimation network DONet is used to build a class-level six-degree-of-freedom pose estimation model. The category and point cloud data of the object are input. The class-level six-degree-of-freedom pose estimation model extracts the geometric features of the object and estimates the rotation and translation of the target object in the camera coordinate system to obtain the six-degree-of-freedom pose of the target object.

[0059] This embodiment also provides a specific application scenario of the above method, specifically:

[0060] like Figure 4 As shown, the method for acquiring the 6D pose of a class-level target object in this embodiment can be used to control the robot to grasp the object using voice, thereby realizing a human-computer interaction system from voice to control.

[0061] This embodiment also relates to a storage medium, in which any of the above-mentioned methods for acquiring 6D pose of a class-level target is stored.

[0062] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for acquiring 6D pose of class-level objects based on language description, characterized in that: The 6D posture acquisition method comprises: Step 1: Construct a two-dimensional object positioning model based on language features and image features; Step 2: Input the image and language description into the two-dimensional positioning model, so as to generate the two-dimensional positioning of the target object in the image according to the language description; Step 3: Back-project the two-dimensional positioning depth image using the camera internal parameters to obtain the point cloud data corresponding to the target area; Step 4: Build a point cloud segmentation model for removing background point clouds, and input the target area point cloud into the point cloud segmentation model to obtain point cloud data of the target object; Step 5: Construct a class-level six-degree-of-freedom pose estimation model for estimating the pose of the object in three-dimensional space, and input the target object point cloud data into the class-level object pose estimation model to obtain the pose of the target object; The step 1 is specifically as follows: Step 1-1: Construct a language-image coupling confidence module; Step 1-2: Calculate the confidence level of fit between the input text and each candidate bounding box image; The language-image coupling confidence module includes: The object subject fit confidence unit fuses the image feature embedding code and the language feature embedding code and sends them to the encoder to regress and generate the fit confidence of the target subject described in the text and the object subject in the picture; The object position fit confidence unit uses the real annotated bounding box information to generate the position embedding code, and predicts the fit confidence of the target position described in the text and the object position in the image through the encoder with the language feature embedding code; The object proximity relationship fit confidence unit uses language feature embedding coding and position embedding coding of neighboring candidate regions to predict the fit confidence of the target-surrounding object relationship described in the text and the object-surrounding object relationship in the image through the encoder; The step 1-1 is specifically as follows: The input image is embedded and coded by extracting image features through the residual neural network ResNet, and the language description is embedded and coded by extracting language features through the bidirectional recurrent neural network Bi-RNN. Then, the object subject fit confidence, object position fit confidence, and object neighbor relationship fit confidence are calculated respectively. The steps 1-2 are specifically as follows: The input image is passed through the region proposal network (RPN) to generate multiple candidate bounding boxes. The image corresponding to each bounding box and the position information of its neighboring bounding boxes are taken as input. The weighted sum of the three confidences obtained in step 1-1 is taken as the sum of the confidences of the bounding box. The candidate bounding box with the highest confidence will be used as the target predicted bounding box, and then the image corresponding to the target predicted bounding box will be sent to Mask-RCNN for further prediction to obtain the category segmentation mask of the object.

2. According to the method for acquiring 6D pose of a class-level target based on language description in claim 1, it is characterized in that: The encoder is a multi-layer perceptron.

3. According to the method for acquiring 6D pose of a class-level target based on language description in claim 1, it is characterized in that: The step 3 is specifically as follows: After obtaining the object bounding box or segmentation mask, the corresponding depth image is intercepted accordingly, and this part of the depth image is back-projected through the camera intrinsic parameters to obtain the corresponding three-dimensional point cloud data in the area.

4. The method for acquiring 6D pose of a class-level target based on language description according to claim 1, characterized in that: The step 4 is specifically as follows: A point cloud segmentation model is built based on the three-dimensional convolutional network 3D-GCN to output the category label of each point cloud for distinction, segment the object point cloud and the background point cloud, and then delete the background point cloud.

5. The method for acquiring 6D pose of a class-level target based on language description according to claim 1, characterized in that: The step 5 is specifically as follows: The class-level pose estimation network DONet is used to build a class-level six-degree-of-freedom pose estimation model. The category and point cloud data of the object are input. The class-level six-degree-of-freedom pose estimation model extracts the geometric features of the object and estimates the rotation and translation of the target object in the camera coordinate system to obtain the six-degree-of-freedom pose of the target object.

6. A computer-readable storage medium, characterized in that: The storage medium stores the class-level target 6D pose acquisition method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Device control method and device based on speech and space object recognition and positioning

    CN109003303A

  • Object six-degree-of-freedom pose estimation method based on color and depth information fusion

    CN111179324A

  • Fine-grained image weak supervision target positioning method based on deep learning

    CN111598155A

  • Point cloud prediction model generation method, pose estimation method and pose estimation device

    CN112652016A

  • Class-level 6D pose and size estimation method and device

    CN113012122A