Method and system for constructing and updating three-dimensional panoramic map and scene graph of robot with body
Through the incremental construction method of three-dimensional panoramic maps based on graph neural network, combined with depth map segmentation and reprojection technology, the changing objects in the environment are detected and updated in real time, and the problem of scene map update of embossed robots in semi-static environments is solved, and the problem of scene map update in semi-static environments is improved. Robot environment understanding and task response capabilities.
Patent Information
- Application Number
- CN202410802130.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2024-12-03
AI Technical Summary
The existing three-dimensional scene map construction technology of embodied robots is difficult to deal with changes in the position and state of objects in semi-static environments, resulting in the continuous decrease in the matching degree of scene maps with the real environment over time, affecting the robot's service capabilities.
Using the incremental construction method of three-dimensional panoramic map and scene graph based on graph neural network, the node and edge characteristics of a single-frame image are obtained, and the categories are aggregated and predicted, and the results of multi-frame image are fused, combined with depth map segmentation and reprojection technology are combined in real time. Detect and update changing objects in the environment to achieve accurate maintenance of three-dimensional panoramic maps.
It improves the embodied robot's understanding and cognitive ability to the environment and task response ability, improves the degree of intelligence, ensures the accuracy and real-time update capabilities of scene maps, and is suitable for complex scenarios such as home services.
Smart Images

Figure CN119068128A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of constructing and updating three-dimensional panoramic maps and scene graphs of embodied robots, and in particular, relates to a method and system for constructing a three-dimensional scene graph of an embodied robot. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] In recent years, large-scale model and embodied intelligence technologies have made rapid progress, and a large number of studies have emerged. AI empowers physical devices to achieve "embodied intelligence" and has aroused global expectations. Scene understanding technology gives robots the ability to perceive the environment, helping them to correctly understand the world and perform tasks efficiently.
[0004] Specifically, robot scene understanding refers to the robot processing data from multiple types of sensors, extracting meaningful semantic information, and thus gaining a comprehensive understanding of the environment. Whether the robot can fully understand the objects in complex scenes, the relationships between objects, and even the relationships between objects and people directly affects the robot's ability to perform service tasks.
[0005] Therefore, improving the environmental perception and understanding capabilities of embodied robots is one of the important research directions of embodied robots. It can help embodied robots better adapt to diverse and complex service scenarios and provide humans with more intelligent and convenient life services.
[0006] However, current scene mapping technologies usually rely on the assumption of a static environment, that is, they assume that the position and shape of objects in the environment will not change. However, the presence of people in a home scene will cause objects to change, and current scene mapping technologies usually have difficulty handling this problem, causing the constructed scene map to deviate from reality over time until it becomes unusable.
[0007] Therefore, how to enable the embodied robot to update and maintain the accuracy of the stored three-dimensional panoramic map in a semi-static scene where the position and state of objects in the scene may change is a key issue that needs to be studied urgently.
[0008] The inventors found in their research that the relevant prior art includes:
[0009] Samuel Looper et al. proposed a method for predicting long-term object instance changes based on a mutable three-dimensional scene graph. They designed a model based on a graph neural network and trained it on the 3RScan dataset. The model can detect the state and position changes of object instances and improve the robot's autonomous cognition of semi-static scene changes. Experimental results show that this method can predict potential changing objects in semi-static environments.
[0010] Based on the ElasticFusion system, Edith Langer et al. combined the advantages of global and local methods and proposed an object change detection algorithm that integrates semantic and planar geometric information. It effectively detects new objects in the 3D reconstruction of indoor environments and uses the global structure determined by 3D semantic information to establish candidate objects. Then, by comparing the isolated geometry with the reference reconstruction provided by the task, the recognition of objects with position changes in semi-static environments is achieved.
[0011] Jiahui Fu et al. proposed an object-level change detection method based on neural description fields, which encodes the potential shapes of objects in the scene that cannot be observed due to view occlusion. At the same time, they proposed a tree search and feature similarity matching algorithm to achieve change detection of objects under view occlusion, and used the shape completion capability of the category-level neural description field to represent the partially observed complete object shape with compact shape encoding. Experimental results show that this method can effectively improve the change detection capability of environmental objects.
[0012] Ue-Hwan Kim et al. proposed an image-based incremental 3D scene graph construction method. First, the local scene graph of the key frame image is extracted. Then, the similarity of object instances in the multi-frame local scene graph is judged by using semantic, position, and color information. Finally, the histogram distance measurement method is used for instance matching to achieve global 3D scene graph fusion update. The authors proved that the proposed method can improve the robot's environmental understanding and task planning capabilities.
[0013] The above-mentioned traditional environmental three-dimensional scene understanding problem is usually solved by SLAM technology. However, this method cannot extract the semantic information of objects in the environment. With the development of deep learning, image understanding technologies, such as object detection, semantic segmentation, instance segmentation, and panoptic segmentation, have enabled embodied robots to have strong image semantic perception capabilities. Panoramic mapping has attracted much attention due to its comprehensive perception capabilities. Unlike mapping methods that only focus on semantics or instances, panoramic mapping can simultaneously distinguish between semantic objects and instance objects in the environment, and has comprehensive perception capabilities for foreground and background objects.
[0014] The 3D scene graph is a refined scene representation model that uses a graph structure to describe the information of objects in the scene and the relationship between objects. It is a simplification and supplement to the metric panoramic map. Existing 3D scene graph construction methods are often based on the constructed scene map, that is, the complete map model is directly input into the end-to-end network to achieve prediction. In comparison, the incremental construction method can not only effectively reduce the computing and storage costs, but also continuously optimize and improve the scene graph based on real-time observation data.
[0015] In practical applications, the incremental three-dimensional scene graph construction algorithm provides more efficient decision-making support for embodied robots. By obtaining the scene information in the current observation in real time, the embodied robot can respond to changes in the environment in a timely manner when performing home service tasks. Objects in home scenes are often moved frequently, resulting in the semi-static characteristics of home scenes. If the embodied robot cannot effectively identify the location or state of objects in the process of home service, then when building the environment map, it will not be able to remove outdated objects in the map, resulting in a continuous decrease in map accuracy. In the long run, the matching degree between the map stored by the embodied robot and the real environment continues to decrease, which seriously affects the service capabilities of the embodied robot in home scenes. Summary of the invention
[0016] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for constructing a three-dimensional scene graph of an embodied robot, which can improve the embodied robot's ability to understand and recognize the environment and its ability to respond quickly to tasks, thereby enhancing the intelligence level of the embodied robot.
[0017] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0018] In a first aspect, a method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot is disclosed, comprising:
[0019] Obtaining the instance object data of the three-dimensional object in the global scene graph, extracting the features of the three-dimensional object and its relationship with respect to the single-frame image, and obtaining the node features and edge features of the single-frame image in the three-dimensional scene graph;
[0020] Input the node features and edge features into the graph neural network, and aggregate the features of the nodes and edges in the graph neural network;
[0021] After aggregation, two layers of MLP are used to predict the node and edge categories of a single frame image, that is, to obtain a local scene graph;
[0022] For the prediction results of node and edge categories of single-frame images at different times, the weights and probabilities of all categories of predicted nodes and edges are stored and fused, and the global 3D scene graph is continuously updated. The image mask corresponding to each node in the 3D scene graph is projected into 3D space, and the predicted node category is used as the semantic label of the object instance in the 3D panoramic map to construct a 3D panoramic map of the environment.
[0023] Based on the constructed global 3D scene graph and 3D panoramic map, objects that need to be updated in the semi-static environment are detected and updated.
[0024] As a further technical solution, the features of the three-dimensional objects and their relationships are extracted to obtain the node features and edge features in the three-dimensional scene graph. The specific steps are as follows:
[0025] 3D object instance data, including the point cloud of the instance object and its 3D bounding box;
[0026] Based on the point cloud and 3D bounding box, directly extract the standard deviation of the point cloud position, the coordinates, size, maximum side length and volume of the 3D bounding box center point;
[0027] The extracted data is processed and then input into the graph neural network, and the output of the graph neural network is subjected to feature aggregation and updated new features.
[0028] As a further technical solution, node features and edge features are input into GNN, wherein a GNN with two message passing layers is used to realize the construction of a local scene graph;
[0029] For each message passing layer of GNN, the node features are first aggregated with the features of the edges connected to it, and then aggregated with the features of its neighboring nodes.
[0030] As a further technical solution, the prediction of node and edge categories includes:
[0031] The network is trained using a loss function, in which a weight factor is used to balance the losses of nodes and edges.
[0032] The image mask corresponding to each node in the 3D scene graph is projected into the 3D space, and the predicted node category is used as the semantic label of the object instance in the 3D panoramic map;
[0033] Output the specific types of nodes and edges.
[0034] As a further technical solution, when detecting objects that need to be updated in a semi-static environment, a depth map segmentation algorithm is used to segment the image into multiple instance masks with continuous textures;
[0035] After segmentation, the image is downsampled to reduce the number of pixels, and on this basis, projection is performed according to the intrinsic and extrinsic parameters of the camera to obtain a sparse point cloud of the object instance under current observation.
[0036] As a further technical solution, after detecting and updating the objects that need to be updated in the semi-static environment, it also includes: using the camera frustum and the three-dimensional bounding box of the objects in the historical map to eliminate objects that cannot be observed in the current viewing angle.
[0037] As a further technical solution, after eliminating objects that cannot be observed from the current viewing angle, the reprojection method is used to eliminate misjudgment caused by object occlusion;
[0038] Afterwards, objects whose positions have changed are detected by comparing the observable historical point cloud and the actual observed point cloud under the current viewing angle in three-dimensional space;
[0039] For objects that are judged to conflict with the observation data at a certain moment, their status is set to pending. Instance objects in the pending state do not participate in the instance fusion in the construction of the 3D panoramic map and scene graph;
[0040] If an instance object is continuously judged to conflict with the real-time observation data of the embodied robot, the object in the pending state is deleted from the map.
[0041] In a second aspect, a system for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot is disclosed, comprising:
[0042] The feature extraction module is configured to: obtain the instance object data of the three-dimensional object in the global scene graph, extract the features of the three-dimensional object and its relationship with respect to the single frame image, and obtain the node features and edge features of the single frame image in the three-dimensional scene graph;
[0043] The feature aggregation module is configured to: input node features and edge features into the graph neural network, and aggregate the features of nodes and edges in the graph neural network;
[0044] The node and edge category prediction module is configured to: predict the node and edge categories of a single frame image through two layers of MLP after aggregation;
[0045] The global 3D scene graph construction module is configured to: store and fuse the predicted results of node and edge categories of single-frame images at different times, and continuously update the global 3D scene graph;
[0046] The updating module is configured to detect and update objects that need to be updated in the semi-static environment based on the constructed global three-dimensional scene graph.
[0047] One or more of the above technical solutions have the following beneficial effects:
[0048] The technical solution of the present invention proposes an incremental construction algorithm of a three-dimensional scene graph based on a graph neural network, obtains the local scene graph in the image, and fuses the prediction results of multiple frames of local scene graphs. Then, an efficient object change detection algorithm in a semi-static environment is used to realize instance-level change detection based on the image geometric segmentation results; finally, the map update technology is used to achieve long-term maintenance of the accuracy of the three-dimensional panoramic map by fusing the change detection results under multiple perspectives. The proposed framework can improve the embodied robot's ability to understand and recognize the environment and its ability to respond quickly to tasks, thereby improving the intelligence of the embodied robot.
[0049] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0051] Figure 1 A framework diagram for constructing an incremental three-dimensional scene graph according to an embodiment of the present invention;
[0052] Figure 2 This is a flow chart of a three-dimensional panoramic map update algorithm according to an embodiment of the present invention;
[0053] Figure 3 A schematic diagram of a camera viewing cone according to an embodiment of the present invention;
[0054] Figure 4 A schematic diagram of a KD-tree according to an embodiment of the present invention;
[0055] Figure 5 Schematic diagram comparing current observation data (green point cloud) and historical map (red point cloud); Figure 6 Three typical scenes in the data set are selected for display in the embodiment of the present invention; Figure 7 For the embodiment of the present invention, a typical scene in the RIO10 dataset is first selected to perform a qualitative analysis on the effect of the proposed map update algorithm. DETAILED DESCRIPTION
[0056] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in this embodiment have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0057] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.
[0058] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0059] Embodiment 1
[0060] This embodiment discloses a method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot, including:
[0061] (a): An algorithm for incrementally building a 3D panoramic map and scene graph based on a graph neural network. This algorithm obtains the local scene graph in the robot’s current observation image, fuses the prediction results of multiple frames of local scene graphs, and builds a 3D panoramic map. The overall process is as follows: Figure 1 As shown;
[0062] (b): An efficient object change detection algorithm in a semi-static environment, which realizes instance-level change detection based on image geometric segmentation results;
[0063] (c): Map update: By fusing the change detection results from multiple perspectives, the long-term maintenance of the 3D panoramic map accuracy is achieved.
[0064] In this implementation example, the 3D panoramic map and the 3D scene graph are different representations of the 3D scene, both describing the features of semantic objects in the 3D scene. The 3D panoramic map can be constructed together with the 3D scene graph.
[0065] The framework proposed in the technical solution of this embodiment can improve the embodied robot's ability to understand the environment and respond quickly to tasks, thereby improving the intelligence of the embodied robot. The overall process of the proposed change detection and map update algorithm is as follows: Figure 2 shown.
[0066] Step 1: In order to accurately predict the categories of nodes and edges in the 3D scene graph, we first need to analyze and process the 3D object instance data, including the point cloud of the instance object and its 3D bounding box, and extract the features of the instance object and its relationship. The node features input to the graph neural network include: the standard deviation of the point cloud position The coordinates of the center point of the 3D bounding box size Maximum side length And volume Since the shapes of objects stored in the map will change as the map is constructed, the node features will also be continuously updated to ensure accurate description of the instance object features. When constructing edge features of a 3D scene graph, the system constructs an adjacency matrix based on the distance between objects and constructs features for neighboring objects. i and edge ij The characteristics are shown below:
[0067]
[0068] In the formula, f p (·) is the PointNet point cloud feature extraction network, [·] is the feature concatenation function, g s (·) is a multi-layer perceptron, P i is the point cloud of the instance object, r ij The features after splicing.
[0069] The standard deviation of the point cloud position, the coordinates, size, maximum side length and volume of the 3D bounding box center point can be directly extracted from the input point cloud and 3D bounding box. These data are processed by equations (1) to (3) and then input into the graph neural network. The output of the graph neural network is the new feature after feature aggregation and update. The structure and loss function of the graph neural network are described in steps 2 and 3.
[0070] Step 2: In order to effectively aggregate the features of nodes and edges in the graph neural network, this example uses a GNN with two message passing layers to build a three-dimensional scene graph. The input of the GNN is the features of nodes and edges obtained in step 1. node and edge The update formula of the feature is as follows:
[0071]
[0072] Where [·] is the feature concatenation function, is the updated node feature, is the neighbor node feature of the node to be updated, is the updated edge feature, g v (·) and g e (·) is a fully connected layer, is the index set of neighbor nodes of node i, FAN(·) is the feature attention network, and the calculation method is as follows:
[0073] FAT(Q,T)=softmax(ga(Q))e T (6)
[0074]
[0075] Where FAT(·) is the feature aggregation function of the query node and the target node, Q and T are the query node feature and the target node feature, respectively. i and τ i are the query node and target node features after splitting, h is the number of split features, g is a (·) is a fully connected layer used for normalization and weight assignment. The purpose of MFAT(·) is to split the features and fuse each feature separately using a multi-head attention mechanism. [·] is the feature aggregation function. Acting on v i , e ij and v j A single-layer perceptron.
[0076] In order to incorporate multi-category node and edge features into feature fusion calculations, the network first aggregates node features with the features of the edges connected to it, and then aggregates them with the features of its neighboring nodes.
[0077] The processing process of this step is to aggregate and update the features of nodes and edges. It should be noted that any node feature is first aggregated with its neighbor node features, and then the current node feature is updated, which is specifically described by equations (4) to (8). The output of the graph neural network is the new feature after feature aggregation and update.
[0078] Step 3: After executing the message aggregation of GNN in step 2, the network predicts the node and edge categories through two layers of MLP, and trains the network using the following loss function:
[0079] L total =λ obj L obj +L pred (9)
[0080] Where, L total is the total loss function of the graph neural network, L obj and L pred Both are Focal loss functions, representing the loss of node and edge category prediction tasks, respectively, obj is the weight factor. Since objects in a home scene usually have relationships with multiple objects around them, the number of edges in the three-dimensional scene graph is much greater than the number of nodes. Therefore, a weight factor is needed to balance the loss of nodes and edges. In order to construct a three-dimensional panoramic map of the environment, this embodiment projects the image mask corresponding to each node in the three-dimensional scene graph into three-dimensional space, and uses the predicted node category as the semantic label of the object instance in the three-dimensional panoramic map. Therefore, after constructing the three-dimensional scene graph, a three-dimensional panoramic map of the environment can be easily constructed.
[0081] The input of the two-layer MLP is connected to the output of the graph neural network in step 2. The output of the MLP based on semantic labels is the specific categories of nodes and edges.
[0082] Step 4: The previous step completes the prediction of the 3D scene graph of a single frame image. However, in order to obtain the 3D scene graph of the complete scene, the prediction results of the single frame image need to be fused. Since the shape of the object instance in the map changes continuously with the movement of the camera, there is an inevitable conflict between the prediction results of the 3D scene graph in different image frames. To solve this problem, for the prediction results at different times, the algorithm adds the weights w of all categories of nodes and edges predicted by the network t and probability μ t Store and continuously update the global 3D scene graph using the following formula:
[0083]
[0084] w t =min(w max ,w t +w t-1 ) (11)
[0085] In the formula, w t-1 and μ t-1 are the weight and probability of the category before updating, w max is the maximum value of the weight, set to 100.
[0086] The previous steps process a single frame image. The purpose of this step is to fuse the prediction results of multiple frames of images to obtain a complete scene graph.
[0087] Step 5: After completing the construction of the three-dimensional panoramic map and scene graph, in order to update the scene map in real time based on the images subsequently observed by the embodied robot, it is necessary to design an image-based object change detection algorithm. The change in the position of an object usually means the movement of the object as a whole, so change detection on instance-level data can undoubtedly improve the accuracy and efficiency of the algorithm. This embodiment uses a depth map segmentation algorithm to segment the image into multiple instance masks with continuous textures, and performs change detection on this basis. Considering that the number of image pixels is too large, this embodiment downsamples the image to reduce the number of pixels, and on this basis, performs projection based on the camera's intrinsic and extrinsic parameters to obtain a sparse point cloud of the object instance under the current observation.
[0088] This step is to detect and update the objects that need to be updated in the semi-static environment after the three-dimensional panoramic map and scene graph are built in the fourth step.
[0089] Step 6: The constructed 3D panoramic map is used as the historical map, and the historical map is updated and corrected: the point cloud data of the historical map under the current camera perspective is obtained. In order to improve the efficiency of the algorithm, this example uses the camera frustum and the 3D bounding box of the objects in the historical map to quickly eliminate objects that cannot be observed under the current perspective. The frustum is used to describe the maximum observation area of the camera, usually a tetrahedron, and is described by six plane equations, such as Figure 3 As shown. For objects in the historical map to be observed from the current camera perspective, the necessary condition is that they are partially located in the viewing cone. Since the 3D bounding box completely surrounds the instance object point cloud, if the bounding box of an object does not intersect with the viewing cone, the object cannot be observed from the current perspective. Based on this premise, the algorithm removes object instances whose bounding boxes do not intersect with the viewing cone in advance to improve the efficiency of the algorithm.
[0090] Step 7: Since objects may block each other, even if part of an object is within the camera's frustum, it cannot be guaranteed that the object will be observed by the camera. Therefore, in order to accurately extract the observable point cloud, this embodiment uses the reprojection method to eliminate the misjudgment caused by object occlusion. However, due to the sparsity of the point cloud, directly projecting the point cloud may cause some actually blocked points to be incorrectly projected into the image. To address this problem, this embodiment appropriately reduces the resolution of the projected image. The pseudo code of the historical map extraction algorithm under the current camera perspective is shown in Table 1.
[0091] Table 1 Historical map extraction algorithm under the current camera perspective
[0092]
[0093]
[0094] Step 8: In three-dimensional space, objects whose positions have changed are detected by comparing the observable historical point cloud and the actual observed point cloud under the current viewing angle. o In this example, the nearest neighbor search algorithm based on KD tree is used to determine whether the object conflicts with the current observation. KD tree is a data structure that efficiently divides data in K-dimensional space. It is essentially a binary tree. Figure 4 As shown. Taking two-dimensional space as an example, the KD tree selects the partition axis in two dimensions in turn, and divides the points on both sides of the axis into the left and right numbers in turn with the partition point as the root node. Then, the above process is recursively executed until each node in the tree contains only one data point, or the maximum depth set in advance is reached.
[0095] In the process of matching the historical map point cloud and the current observation point cloud, for each target point of the instance object in the historical map The algorithm first starts from the root node of the KD tree constructed by the actual observation point cloud, compares the target point with the nodes in the tree, and decides to traverse the left subtree or the right subtree according to the spatial distance between the nodes in the tree and the target point, and continuously recurses the process until it reaches a leaf node. Then, after reaching the leaf node, the algorithm backtracks upward to determine whether the backtracked node is closer to the target point. By repeating the above process, the algorithm can efficiently find the spatial point in the actual observation point cloud that is closest to the target point. This embodiment traverses P o All spatial points stored in the object are searched and the search radius is set to determine whether the target point is successfully matched. If the number of successfully matched spatial points in the instance object point cloud reaches a certain scale, it is considered that the instance object does not conflict with the current observation and that the object has not changed its position. Otherwise, it is considered that the object has changed its position, for example Figure 5In the figure, the chair (red dot) in the historical map has almost no successful match with the spatial point (green dot) in the current observation point cloud. The algorithm detects that the object conflicts with the current observation and is a potential change object. The process of the conflict detection algorithm is shown in Table 2.
[0096] Table 2 Conflict detection algorithm
[0097]
[0098]
[0099] Step 9: In order to improve the robustness of map updating, it is necessary to combine the change detection results of multiple frames of images to comprehensively judge whether the objects in the three-dimensional panoramic map have changed. For objects that are judged to conflict with the observation data at a certain moment, this embodiment does not directly delete them from the map, but sets their status to pending. Instance objects in the pending state do not participate in the instance fusion in the construction of the three-dimensional panoramic map and the scene graph. During the camera movement process, the algorithm continuously counts the number of times the instance objects in the historical map are detected as conflicting and non-conflicting, and the statistical results are used to comprehensively determine whether the instance objects are deleted from the map. If the instance object state is only maintained as pending without deleting the object, the storage volume of the three-dimensional panoramic map will continue to increase when the embodied robot is running for a long time, affecting the efficiency of the embodied robot in performing tasks such as positioning and path planning. Therefore, timely cleaning of instance objects in the pending state is crucial to maintaining the availability of the map. As for whether to delete objects set to the pending state from the map, the judgment criteria adopted by this embodiment are: a certain instance object is continuously judged to conflict with the real-time observation data of the embodied robot.
[0100] In steps 5 to 9 of this embodiment, these steps can identify objects that have changed in a semi-static environment and update the positions of these objects.
[0101] This example first verifies the 3D scene graph construction algorithm in the framework on the RIO10 data. This example selects three typical scenes in the dataset for demonstration, such as Figure 6 From the experimental results, the example method of this embodiment can extract the relationship between instance objects on the basis of constructing a three-dimensional panoramic map, construct a three-dimensional scene graph of the environment, and realize the refinement of the panoramic metric map.
[0102] This example first selects typical scenes in the RIO10 dataset to qualitatively analyze the effect of the proposed map update algorithm. Figure 7As shown. This example first runs the original sequence in the data set, and then runs the new sequence collected after a period of time after the object position changes. This example selects SceneGraphFusion as the comparison method. Figure 7 The experimental results show that the 3D panoramic map update algorithm proposed in this example can detect most of the object instances with position changes in the historical map (such as Figure 7 The red circle shows the effectiveness of the proposed object change detection and map updating algorithm.
[0103] This example analyzes the average running time of each module of the proposed change detection algorithm. By running the algorithm in all 10 scenarios of the RIO10 dataset, this example calculates the average running time of each module, and the results are shown in Table 3.
[0104] Table 3 Time analysis of each module of change detection algorithm
[0105]
[0106] This example uses the Chamfer Distance (CD) to evaluate the proposed map update algorithm. The role of the Chamfer Distance is to calculate the similarity between two point clouds. The smaller the Chamfer Distance between the constructed map and the real map, the higher the accuracy of the constructed map:
[0107]
[0108] Where S1 and S2 represent the two point clouds to be evaluated, and x and y represent the spatial points in the two point clouds.
[0109] This example further quantitatively verifies the effect of the proposed map updating method on the RIO10 dataset, as shown in Table 4. The experimental results show that the example method of this embodiment achieves better mapping results on most sequences. From the mean value of the chamfer distance, the accuracy of the example method of this embodiment is improved by 1.64% compared with SceneGraphFusion. Since the number of objects that change in different sequences is different, the accuracy improvement of the example method of this embodiment on different sequences varies. For example, in sequences 4 and 9, the example of this embodiment has achieved an improvement of more than 5mm. This is because the number of objects that change in this sequence is large, which can better reflect the advantages of the example method of this embodiment.
[0110] Table 4 Qualitative results of map updating in RIO10 dataset
[0111]
[0112] Embodiment 2
[0113] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0114] Embodiment 3
[0115] The purpose of this embodiment is to provide a computer-readable storage medium.
[0116] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.
[0117] Embodiment 4
[0118] The purpose of this embodiment is to provide a system for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot, including:
[0119] The feature extraction module is configured to: obtain the instance object data of the three-dimensional object in the global scene graph, extract the features of the three-dimensional object and its relationship with respect to the single frame image, and obtain the node features and edge features of the single frame image in the three-dimensional scene graph;
[0120] The feature aggregation module is configured to: input node features and edge features into the graph neural network, and aggregate the features of nodes and edges in the graph neural network;
[0121] The node and edge category prediction module is configured to: predict the node and edge categories of a single frame image through two layers of MLP after aggregation;
[0122] The global 3D scene graph construction module is configured to: store and fuse the predicted results of node and edge categories of single-frame images at different times, and continuously update the global 3D scene graph;
[0123] The updating module is configured to detect and update objects that need to be updated in the semi-static environment based on the constructed global three-dimensional scene graph.
[0124] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.
[0125] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0126] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot, characterized in that: include: Obtaining the instance object data of the three-dimensional object in the global scene graph, extracting the features of the three-dimensional object and its relationship with respect to the single-frame image, and obtaining the node features and edge features of the single-frame image in the three-dimensional scene graph; Input the node features and edge features into the graph neural network, and aggregate the features of the nodes and edges in the graph neural network; After aggregation, two layers of MLP are used to predict the node and edge categories of a single frame image, that is, to obtain a local scene graph; For the prediction results of node and edge categories of single-frame images at different times, the weights and probabilities of all categories of predicted nodes and edges are stored and fused, and the global 3D scene graph is continuously updated. The image mask corresponding to each node in the 3D scene graph is projected into 3D space, and the predicted node category is used as the semantic label of the object instance in the 3D panoramic map to construct a 3D panoramic map of the environment. Based on the constructed global 3D scene graph and 3D panoramic map, objects that need to be updated in the semi-static environment are detected and updated.
2. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: Extract the features of 3D objects and their relationships to obtain node features and edge features in the 3D scene graph. The specific steps are as follows: 3D object instance data, including the point cloud of the instance object and its 3D bounding box; Based on the point cloud and 3D bounding box, directly extract the standard deviation of the point cloud position, the coordinates, size, maximum side length and volume of the 3D bounding box center point; The extracted data is processed and then input into the graph neural network, and the output of the graph neural network is subjected to feature aggregation and updated new features.
3. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: The node features and edge features are input into the GNN, where the GNN with two message passing layers is used to construct the local scene graph; For each message passing layer of GNN, the node features are first aggregated with the features of the edges connected to it, and then aggregated with the features of its neighboring nodes.
4. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: When predicting node and edge categories, it includes: The network is trained using a loss function, in which a weight factor is used to balance the losses of nodes and edges. The image mask corresponding to each node in the 3D scene graph is projected into 3D space, and the predicted node category is used as the semantic label of the object instance in the panoramic map; Output the specific types of nodes and edges.
5. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: When detecting objects that need to be updated in a semi-static environment, a depth map segmentation algorithm is used to segment the image into multiple instance masks with continuous textures; After segmentation, the image is downsampled to reduce the number of pixels, and on this basis, projection is performed according to the intrinsic and extrinsic parameters of the camera to obtain a sparse point cloud of the object instance under current observation.
6. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: After detecting and updating the objects that need to be updated in the semi-static environment, the following steps are also included: using the camera frustum and the three-dimensional bounding box of the objects in the historical map to eliminate objects that cannot be observed from the current perspective.
7. The method for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot as claimed in claim 1, characterized in that: After eliminating objects that cannot be observed from the current viewing angle, the reprojection method is used to eliminate misjudgments caused by object occlusion; Afterwards, objects whose positions have changed are detected by comparing the observable historical point cloud and the actual observed point cloud under the current viewing angle in three-dimensional space; For objects that are judged to conflict with the observation data at a certain moment, their status is set to pending. Instance objects in the pending state do not participate in the instance fusion in the construction of the 3D panoramic map and scene graph; If an instance object is continuously judged to conflict with the real-time observation data of the embodied robot, the object in the pending state is deleted from the map.
8. A system for constructing and updating a three-dimensional panoramic map and scene graph of an embodied robot, characterized in that: include: The feature extraction module is configured to: obtain the instance object data of the three-dimensional object in the global scene graph, extract the features of the three-dimensional object and its relationship with respect to the single frame image, and obtain the node features and edge features of the single frame image in the three-dimensional scene graph; The feature aggregation module is configured to: input node features and edge features into the graph neural network, and aggregate the features of nodes and edges in the graph neural network; The node and edge category prediction module is configured to: predict the node and edge categories of a single frame image through two layers of MLP after aggregation; The global 3D scene graph construction module is configured to: store and fuse the predicted results of node and edge categories of single-frame images at different times, and continuously update the global 3D scene graph, project the image mask corresponding to each node in the 3D scene graph into the 3D space, and use the predicted node category as the semantic label of the object instance in the 3D panoramic map to construct a 3D panoramic map of the environment; The updating module is configured to detect and update objects that need to be updated in the semi-static environment based on the constructed global three-dimensional scene graph and the three-dimensional panoramic map.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are performed.
