Efficient scene representation method for task-oriented robot

Abstract

This invention relates to an efficient scene representation method for task-oriented robots, comprising the following steps: acquiring RGB image frames and their depth information from different perspectives captured by the robot; performing panoramic segmentation on each image frame to obtain 2D semantic information and 2D instance information, and projecting them onto a 3D mesh space based on the corresponding depth information to obtain 3D semantic information and 3D instance information; fusing the 3D semantic information and 3D instance information of each image frame into the same world coordinate system based on the inter-frame camera pose, and then using the TSDF method to perform incremental mapping based on the fused 3D semantic information and 3D instance information, and updating voxels based on the semantic information and instance information during mapping to obtain a panoramic scene point cloud; and predicting a 3D scene graph based on the panoramic scene point cloud, using all instances as nodes and the relationships between instances as edges. This invention can generate a 3D semantic scene graph that efficiently represents a scene.

Description

Technical Field

[0001] This invention relates to the field of 3D scene reconstruction technology, and in particular to an efficient scene representation method for task-oriented robots. Background Technology

[0002] Efficient representation of 3D scenes is a crucial foundation for task-oriented intelligent systems to correctly perceive and understand their environment and successfully complete human-issued commands. Currently, semantic mapping systems rely on sensors to acquire 3D information about the surrounding environment, extract effective features or semantic information, and finally represent it in a data form that robots can understand—a 3D semantic map. This is the main process of scene perception in intelligent systems, and its key foundations mainly include the reconstruction of 3D scene geometric information and the perception and understanding of semantic information. The former primarily involves constructing a 2D planar map, a sparse 3D point cloud map, or a dense point cloud map. While these maps represent the coordinates of point clouds and camera poses in space, for intelligent systems, the only effective information is whether a certain location is passable; they have no further direct or effective use value.

[0003] The paper "Incremental 3D Semantic Scene Graph Prediction from RGB Sequences" proposes a real-time incremental 3D scene graph construction framework for incrementally building consistent 3D semantic scene graphs from RGB image sequences. The system uses the pose of the input frame to project point entity labels from the global point cloud map onto the image plane, constructing a reference entity mask. The consistency of the corresponding labels is evaluated by comparing the predicted 2D label mask of the input frame with the projected reference entity mask, followed by label fusion and confidence score updates. The fused and updated entity masks need to be remapped back into 3D space before the global map can be updated. During this process, due to the limited viewpoint of the current frame, the constructed reference entity mask may only represent a portion of the entities in 3D space. After adjusting for label consistency with the predicted mask of the current frame and projecting back into 3D space, local entity labels may change, leading to a decrease in the prediction quality of global point cloud map instances.

[0004] Furthermore, the input node and edge features of the scene graph prediction network in this system only contain image and geometric features. However, because the global point cloud is sparse or incomplete during reconstruction, the effectiveness of the extracted geometric features cannot be guaranteed, severely affecting the accurate prediction of scene graph node and edge categories. In both 2D images and 3D global maps, the semantic information of entities has already been predicted and preliminarily optimized; these semantic features are not effectively utilized in scene graph generation and message passing. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an efficient scene representation method for task-oriented robots, which can solve the above-mentioned problems and generate a three-dimensional semantic scene map that efficiently represents the scene.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide an efficient scene representation method for task-oriented robots, comprising the following steps:

[0007] Acquire RGB image frames and their depth information from different perspectives captured by the task-oriented robot;

[0008] Panoramic segmentation is performed on each image frame to obtain 2D semantic information and 2D instance information, and 3D semantic information and 3D instance information are obtained by projecting the corresponding depth information onto the 3D mesh space.

[0009] Based on the inter-frame camera pose, the 3D semantic information and 3D instance information of each image frame are fused into the same world coordinate system. Then, based on the fused 3D semantic information and 3D instance information, the TSDF method is used to perform incremental mapping. During the mapping process, voxel fusion and updates are performed based on the semantic information and instance information to obtain the panoramic scene point cloud.

[0010] Based on the panoramic scene point cloud, a 3D scene map is predicted by using all instances as nodes and the relationships between instances as edges.

[0011] Furthermore, the semantic information includes semantic categories and semantic confidence levels, wherein the semantic categories include several Stuff semantic categories and several Things semantic categories; and the instance information includes instance ID and instance confidence levels.

[0012] Furthermore, the voxel fusion update based on semantic information and instance information during graph construction includes:

[0013] For any voxel of the Stuff semantic category, the ratio of the Stuff semantic category with the most predictions to the total number of predictions of its semantic categories is used as the semantic category proportion value.

[0014] If the semantic category percentage of a voxel is less than or equal to a set threshold, the semantic category of the voxel is set to Void; otherwise, its TSDF value is updated based on its semantic category percentage.

[0015] Furthermore, the TSDF values ​​of the Stuff semantic category voxels are fused and updated according to the following formula:

[0016]

[0017] Wherein, TSDF(v,label) represents the TSDF value of voxel v of the Stuff semantic category. n is the number of times voxel v is predicted to be the semantic category c of Stuff. l θ represents the total number of times the semantic category of voxel v has been predicted, and θ is the set threshold.

[0018] Furthermore, when calculating the proportion of the semantic category, the semantic confidence is used to adjust the weight of the corresponding semantic category.

[0019] Furthermore, the voxel fusion and update based on semantic information and instance information during graph construction also includes:

[0020] Based on the fused 3D semantic information and 3D instance information, target association is performed on voxels of each Things semantic category in each image frame to obtain multiple global instance IDs;

[0021] For any voxel of the Things semantic category in the current image frame, update its instance ID to the global instance ID and calculate its cumulative weight value based on instance confidence.

[0022] If the cumulative weight value of the current voxel is greater than its instance confidence, then the voxel is fused with the corresponding instance in the point cloud; otherwise, it is treated as a new instance.

[0023] Furthermore, the TSDF values ​​of the Things semantic category voxels are fused and updated according to the following formula:

[0024]

[0025] Among them, TSDF i (v,insID) and TSDF i-1 (v, insID) represent the TSDF values ​​of voxels v in the current and previous Things semantic categories, respectively, and insID i (v) represents the updated instance ID of the current voxel v. and Let represent the cumulative weight value and the current weight value of the current voxel v, respectively. confidence ID This represents the instance confidence of the current voxel v.

[0026] Furthermore, the prediction to obtain a 3D scene map includes:

[0027] For any node, extract its 3D and 2D features respectively, and generate semantic label features based on its semantic information;

[0028] Using 2D and 3D features as input to Value, and semantic label features as input to Query and Keys, multi-attribute feature fusion is performed based on a multi-head attention mechanism.

[0029] The fused features are fed into a graph prediction network to obtain a 3D scene graph.

[0030] Furthermore, the inter-frame camera pose is the change in camera pose between any two adjacent image frames estimated based on SLAM.

[0031] Furthermore, the depth information is acquired by a sensor or obtained from image depth estimation.

[0032] Beneficial effects

[0033] Due to the adoption of the above technical solutions, this invention has the following advantages and positive effects compared with the prior art: Based on TSDF and Voxblox, this invention achieves incremental mapping by fusing inter-frame semantics for the background Stuff class and inter-frame consistency constraints for the target object Things class, realizing multi-view instance association and semantic consistency fusion. On this basis, it combines image, geometry and predicted semantic label features to complete the construction and updating of entities and relationships between entities in the map, and outputs a three-dimensional semantic scene map that can efficiently represent the scene. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0035] Figure 2 This is a structural diagram of the multi-attribute feature fusion module according to an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0037] The embodiments of the present invention relate to an efficient scene representation method for task-oriented robots, such as... Figure 1 As shown, it includes the following steps:

[0038] First, the RGB sequence captured by the robot is sent to the 2D semantic module. The single-view 2D image is segmented in a panoramic manner to obtain pixel-level semantic and instance information. At the same time, it is projected into the three-dimensional grid space according to the depth information to obtain the 3D semantic and instance information of the current view (the depth information can be directly captured by the sensor or obtained by image depth estimation).

[0039] Then, the projection results are input into the 3D reconstruction and semantic perception module. Based on the inter-frame camera pose estimated by SLAM, the 3D semantics and instance information of the current viewpoint are integrated into the same world coordinate system to realize multi-view 3D semantic / instance fusion.

[0040] Based on this, the 3D scene graph prediction module uses instances (same entities) in the fused 3D map as graph nodes to construct the initial scene graph. The graph convolutional network is used to pass messages between image, geometric and predicted semantic label features to predict the node categories and the relationships between entities, and update the 3D semantic scene graph.

[0041] The 2D semantic module is used to adjust the number of channels in the segmentation head of the transformer-based panoramic segmentation framework K-Net to match the number of categories in ScanNetv2. Then, it is trained using instance segmentation and semantic segmentation data from the dataset to achieve fine-tuning, resulting in a model capable of performing high-precision panoramic segmentation of indoor images.

[0042] The 3D reconstruction and semantic perception module utilizes incremental mapping based on the Truncated Signed Distance Function (TSDF). During the mapping process, it adds semantic and instance information for each raster to complete the fusion and updating of semantic attributes, establishing a 3D panoramic semantic mapping system that perceives the geometric and semantic information of the 3D scene in real time. To efficiently achieve 3D reconstruction and panoramic semantic perception, this module is based on Voxblox, employing a 2D-to-3D approach. A ray is projected from the camera center (optical center) through all image pixels to their corresponding 3D voxels, and then the TSDF algorithm is used to update the corresponding voxels.

[0043] In addition to completing 3D reconstruction, the main function of this module is to achieve semantic and temporal consistency of instances across different viewpoints, specifically including two operations:

[0044] First, for the inter-frame semantic fusion of the background Stuff class: the Merged fusion method in the Voxblox codebase has been optimized, and for each voxel, additional attributes besides the TSDF value and its weights have been added: semantic category, semantic confidence, instance ID, and instance confidence {label, confidence}. l ,insID,confidence ID This is used to record the semantic category and instance ID of the scene. For uncountable background semantic categories like sky and floor (called Stuff classes), there is only one such category in the entire scene. For example, even if the floor is occluded by other objects and divided into several non-contiguous pixel pieces in the image, it is still essentially the same floor, and the instance IDs of these floor pieces should always be the same. Therefore, the fusion of different perspectives of the Stuff class is essentially the integration of semantic categories, without the distinction of instance IDs. Thus, for each voxel v, if its label(v) belongs to the Stuff class, then insID(v) = label(v), confidence l =confidence ID Its integration and update are performed according to:

[0045]

[0046] in, It is the number of times the voxel is predicted to be class C, n l This represents the sum of the number of times all semantic classes have been predicted for that voxel. If the ratio of the most frequently predicted semantic class to the total number of predictions does not reach a given threshold θ, then the semantic class of that voxel is assigned as void, indicating that the segmentation model cannot successfully recognize the object. In addition, to further mitigate the adverse effects of the instability or performance limitations of 2D image panoramic segmentation networks on 3D semantic perception, Equation 1... There is another confidence-based calculation method available. At this point... Instead of directly calculating the number of times a voxel is predicted as class C, it calculates the confidence corresponding to each voxel being predicted as class C. l Accumulate the sum. Meanwhile, n... l This is the sum of all categories corresponding to that voxel.

[0047] Second, regarding the inter-frame consistency constraint for the "Things" class of target objects: Before integrating the semantic prediction results from different 2D perspectives into a unified 3D coordinate system space, it is necessary to ensure that they belong to the same object. Unlike semantic categories, which are a global concept, the instance IDs of each target object are independent in the images observed by the camera at each time step, because these IDs themselves have no actual meaning; they are merely sequence numbers used to distinguish different instances. Therefore, when a new frame of point cloud is obtained, it is first necessary to establish associations between the instances in the current perspective and the target objects in the existing map to be updated before further updating the semantic categories.

[0048] The data association task of matching the segmented results of objects in the `Things` class with an existing reconstructed map can be formulated as a Linear Assignment Problem (LAP). Assume there are `n` instance segmentation fragments, each with a set of 3D points associated with its instance ID, and the confidence of the corresponding predicted semantic results of the segmentation model is known. Also, assume there are `m` traceable and similar target object point sets in the constructed 3D model. In 2D image panoramic segmentation, there is no overlap between segmentation fragments, and the segmentation model predicts that each instance object is segmented as a whole. Therefore, it can be assumed that each segmentation fragment can only come from a single target, and that only one target in 3D space is associated with it. With these assumptions, the matching problem can be formulated as an `n×m` matrix, where each row corresponds to a segmentation fragment, each column corresponds to an instance in 3D space, and each element in the matrix represents the similarity between the current viewpoint segmentation fragment and the instance object in the existing 3D model. The optimal solution to this problem can obviously be obtained by matching each row to each column and maximizing the sum of all matching possibilities.

[0049] Meanwhile, since the number of instance objects existing in the existing 3D reconstruction model is unknown, a mechanism for determining and generating new instance objects that have not appeared before is needed. Based on the aforementioned LAP (Layered Object Algorithm) to implement Things object association, the new object generation mechanism is very simple to implement. It only requires determining whether the similarity (IoU) between the current object segment and its best matching instance is lower than a certain threshold. If yes, the object in the 3D space corresponding to the object segment will be treated as a new instance object and assigned a new global instance ID. To quickly find the optimal solution, the Hungarian algorithm is used to solve this LAP problem. Many variations of this method have been widely used in multi-object tracking and can generally complete matching in O(n³) time.

[0050] After establishing the association between Things class objects by obtaining the best match, the local instance IDs from the current viewpoint are converted into global instance IDs. Based on this, the segmentation results from the current viewpoint can be projected and fused into existing 3D voxels for updating. If the voxel itself has color texture information, a weighted average operation is performed on the three channel values. However, semantic categories and instance IDs are different; they must be discrete and independent. That is, there is no situation like depth or color channels, where smaller numerical value differences indicate greater similarity, or where rounding can be used for approximation. Therefore, a confidence-based semantic fusion method similar to that used in the Stuff class is needed to update the semantic information of the already associated Things class objects.

[0051] It is known that each voxel stores {label, confidence} l ,insID,confidence ID For semantic attributes such as}, the corresponding update method is as follows:

[0052]

[0053] in,

[0054]

[0055] in If the instance ID of a newly added voxel corresponding to the current viewpoint is the same as that of the corresponding voxel in the existing map, the weight is accumulated; otherwise, it is decreased. The instance ID of the voxel is changed to the ID corresponding to the current viewpoint only if the accumulated weight is less than the weight of the instance ID in the current viewpoint. Once the instance ID is updated, the semantics of the voxel are also updated to the semantic category corresponding to that instance ID. At this point, all semantic information related to voxels in the 3D scene is updated.

[0056] 3D scene graph prediction module

[0057] This module constructs an initial scene graph based on the obtained panoramic scene point cloud, using all 3D entities (instances) as nodes N and the relationships between them as edges E. 2D texture information and 3D geometric information are integrated to construct features for nodes and edges with multiple attributes. These features and the original graph are then fed into a semantic scene graph generation network for message passing to predict the final node semantics and the relationships between them.

[0058] Specifically, for each node N i ∈N, firstly, PointNet is used to encode the point cloud segmentation as its 3D geometric features f 3dFurthermore, from the keyframes of the observable viewpoints of this instance, the 2D features f of the node are obtained using a VGG or ResNet image encoder. 2d In addition to these, each node's attributes also include semantic information about the instance obtained from the 2D semantic and 3D panoramic mapping modules, which is encoded into semantic label features f. sem (In practice, semantic information is a number corresponding to the text, which can be encoded as a string of 0s and 1s in binary format.) These attributes constitute the initial node features [f] 2d ,f 3d ,f sem It will then be fed into a graph neural network (GNN) to interact and update based on the features of its associated nodes.

[0059] Current scene graph generation systems only input the image or geometric features of the target entity into the network and predict the semantics of the nodes and the relationship categories of the edges. In contrast, this invention makes full use of the 2D and 3D semantic labels predicted in the early stage of the system, embeds the estimated entity labels as semantic label features of the nodes, and feeds them together with the image and geometric features into the graph prediction network to predict and update the final node category and edge relationship category.

[0060] Since the integrity of 3D reconstruction and the accuracy of semantic perception cannot be guaranteed, image, geometric, and semantic label features are not entirely reliable. Therefore, this invention implements a cross-attribute multi-head attention module to effectively fuse these different features. (Appendix) Figure 2 The structure of this module is illustrated. A multi-head attention mechanism is used to generate a mapping between 2D and 3D appearance features and semantic label features, which can provide additional attention to specific regions from different attributes. Here, semantic label features are used as inputs for the query and keys, while 2D and 3D appearance features are used as inputs for the values. Notably, common cross-attention modules typically use the same inputs for keys and values, while this module uses different feature inputs to unite different feature spaces. In this module, attention scores are obtained based on semantic label features, and then effective information is extracted from the appearance features. Finally, the reweighted image features and geometric features are concatenated and convolved with the semantic label features to achieve reasonable feature fusion.

[0061] Based on nodes and their positional relationships, this invention uses edges E as connections between objects to construct an initial adjacency graph G = (N, E). If node N... i N j If ∈N are close to each other, then there exists an edge E. i→j ∈E. Using the two nodes N connected. i and N jCalculate the node attributes of each edge E i→j Features include the offset of the centroid O, the boundary dimensions (from the 3D bounding box b), and the rotation angle R. i→j Specifically represented as E i→j =[o j -o j ,log(b j -b i ),R i→j The initial features of the nodes were obtained using an image and point cloud encoder, while a one-hot encoding method was introduced to embed semantic label features f. sem For edges, initial features are extracted by projecting the properties of the connections onto the latent space using a Multilayer Perceptron (MLP). Then, the features of all nodes and edges are jointly updated by a Graph Neural Network (GNN) that aggregates messages from neighbors. It consists of two GRU (Gated Recurrent Unit) structures shared by all nodes and edges. According to message passing, if a node's low-level features are updated, then all high-level features of that node, its direct neighbors, and their edge features are affected.

[0062] To quantitatively verify the system performance, experiments were conducted on the 3RScan dataset for scene graph prediction. Table 1 presents the experimental results. Compared with other existing scene graph construction methods (such as 3DSSG, SGFN, and MonoSSG), this invention significantly improves the recall (Top 1) of nodes (entities, 20 classes in total), edge relations (predicates, 8 classes in total), and triples (node ​​1 - edge - node 2) in terms of recall (Top 1).

[0063] Table 13 Comparison of 3D semantic scene graph prediction results on the RScan dataset

[0064]

[0065] References:

[0066] [1]Wu SC,Tateno K,Navab N,et al.Incremental 3d semantic scene graphprediction from rgb sequences[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:5064-5074.

[0067] [2]Wald J,Dhamo H,Navab N,et al.Learning 3d semantic scene graphsfrom 3d indoorreconstructions[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2020:3961-3970.

[0068] [3]Wu S C,Wald J,Tateno K,et al.Scenegraphfusion:Incremental 3d scenegraph prediction from rgb-d sequences[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:7515-7525.

Claims

1. An efficient scene representation method for task-oriented robots, characterized in that, Includes the following steps: Acquire RGB image frames and their depth information from different perspectives captured by the task-oriented robot; Panoramic segmentation is performed on each image frame to obtain 2D semantic information and 2D instance information, and 3D semantic information and 3D instance information are obtained by projecting the corresponding depth information onto the 3D mesh space. Based on inter-frame camera pose, 3D semantic information and 3D instance information from each image frame are fused into the same world coordinate system. Then, incremental mapping is performed using the TSDF method based on the fused 3D semantic and instance information. During mapping, voxel fusion updates are performed based on the semantic and instance information to obtain a panoramic scene point cloud. The semantic information includes semantic categories and semantic confidence scores. The semantic categories include several "Stuff" semantic categories and several "Things" semantic categories. The instance information includes instance IDs and instance confidence scores. The voxel fusion updates based on the semantic and instance information during mapping include: For any voxel of the Stuff semantic category, the ratio of the Stuff semantic category with the most predictions to the total number of predictions of its semantic categories is calculated as the semantic category proportion value. When calculating the semantic category proportion value, the semantic confidence is used to adjust the weight of the corresponding semantic category. If the semantic category percentage of a voxel is less than or equal to a set threshold, the semantic category of the voxel is set to Void; otherwise, its TSDF value is updated based on its semantic category percentage. The TSDF value of a Stuff semantic category voxel is updated based on the following formula: in, voxels representing the semantic category of Stuff TSDF value, It is a voxel Predicted as a Stuff semantic category Number of times, Representative voxels The total number of times the semantic categories were predicted. It is about setting a threshold; Based on the fused 3D semantic information and 3D instance information, target association is performed on voxels of each Things semantic category in each image frame to obtain multiple global instance IDs; For any voxel of the Things semantic category in the current image frame, update its instance ID to the global instance ID and calculate its cumulative weight value based on instance confidence. If the cumulative weight value of the current voxel is greater than its instance confidence, then the voxel is fused with the corresponding instance in the point cloud; otherwise, it is treated as a new instance. The TSDF value of the Things semantic category voxels is fused and updated according to the following formula: in, and The voxels representing the current and previous Things semantic categories, respectively. TSDF value, Indicates the current voxel The updated instance ID, and Representing the current voxel The cumulative weight value and the current weight value, and , Indicates the current voxel Instance confidence; Based on the panoramic scene point cloud, a 3D scene map is predicted by using all instances as nodes and the relationships between instances as edges.

2. The method according to claim 1, characterized in that, The prediction obtains a 3D scene map, including: For any node, extract its 3D and 2D features respectively, and generate semantic label features based on its semantic information; Using 2D and 3D features as input to Value, and semantic label features as input to Query and Keys, multi-attribute feature fusion is performed based on a multi-head attention mechanism. The fused features are fed into a graph prediction network to obtain a 3D scene graph.

3. The method according to claim 1, characterized in that, The inter-frame camera pose is the change in camera pose between any two adjacent image frames estimated based on SLAM.

4. The method according to claim 1, characterized in that, The depth information is acquired by sensors or obtained from image depth estimation.

Images