Method and system for spectral graph sparsification in 3D scene graphs

Abstract

The present invention provides a method and system for spectral graph sparsification in a 3D scene graph, the method comprising: sensor data collection; scene graph construction; feature analysis and feature extraction; construction of a Laplacian matrix of the scene graph; calculation of eigenvalues ​​and eigenvectors; spectral sparsification; and verification of the connectivity and validity of the sparse graph. The present invention solves the technical problems of node redundancy in large-scale three-dimensional scene graphs, which leads to a significant increase in the scale of calculation and storage, and high complexity of the scene graph.

Description

Technical Field

[0001] The present invention relates to the fields of three-dimensional scene graphs, real-time positioning and map construction, and scene understanding, and in particular to a method and system for sparsely generating a spectral graph in a 3D scene graph. Background Art

[0002] A 3D scene graph is a data structure used to represent a 3D spatial environment. It organizes spatial information in a hierarchical manner, thereby providing a rich semantic understanding of the environment for the robotic system. The nodes represent spatial concepts at different levels of abstraction, while the edges represent the relationships between these concepts. This data structure can capture information from low-level geometry to high-level semantics, such as objects, places, rooms, buildings, etc. The geometric layer is usually composed of a 3D mesh or voxel model to represent the geometry and spatial layout of the environment. The semantic layer gives semantic labels to objects in the environment based on the geometric layer, such as "chair", "table", etc. These labels help the robot understand the objects in the environment and their functions. The topological layer represents the topological structure of the environment by dividing the space into regions or "places". These places may represent rooms, open areas, or other meaningful spatial partitions. The object layer consists of a set of nodes, each of which represents a specific object in the environment and has a semantic label, center of mass, and bounding box. The place layer represents the free space in the environment in the form of a graph, where nodes represent the location of barrier-free space and edges represent the accessibility between places. The room layer further divides the areas in the venue layer into rooms, and each room node may contain a semantic label, indicating the type or function of the room. At a higher level, the building layer organizes rooms into larger structures such as buildings or complexes, which helps to understand the environment at a more macro level.

[0003] While existing 3D scene graphs provide powerful representations of environments for robotics and automation systems, they still have some limitations. The construction and maintenance of 3D scene graphs can require a lot of computational resources, especially in real-time applications. As the size of the environment increases, the computational burden increases significantly, so it is necessary to find a balance between the sparsity of the graph (reducing computation and storage requirements) and density (preserving environmental details).

[0004] The existing invention patent application document "A method for constructing a three-dimensional scene graph based on hierarchical representation and semantic modeling" with publication number CN118397204A, the existing method includes: 1) data collection to form a metric-semantic voxel map; 2) constructing the first layer of the scene graph: a network layer for representing basic information of the environment; 3) constructing the second layer of the scene graph: an object and agent layer for obtaining the drone posture graph and the spatial layout and characteristics of objects in the environment; 4) constructing the third layer of the scene graph: a location layer for compressing voxels: 5) constructing the fourth layer of the scene graph: a scene layer for classifying and marking spaces; 6) constructing the fifth layer of the scene graph: building nodes connected to all rooms; 7) performing closed-loop detection and geometric verification on the constructed 3D scene graph. And the existing invention patent application document "A method and system for autonomously constructing and maintaining an indoor environment model" with publication number CN116358555A, the existing method includes: using graph structures of different levels to describe indoor scenes, and constructing a three-dimensional scene graph environment model of a mobile robot; using a hierarchical path planning framework to generate a global navigation path and a local navigation path of a neighborhood; determining the target point to be explored, calculating the information gain and path cost from the current position of the mobile robot to the target point to be explored, selecting the optimal target point, searching for a collision-free path from the current position to the optimal target point, and continuously iterating until the conditions are met to complete the autonomous construction of the indoor environment model; using a partially observable cyclic reinforcement learning algorithm to solve the optimal action of the mobile robot under the current observation, obtain the changes of multiple objects in the indoor environment, and maintain and update the indoor environment model.

[0005] However, the two aforementioned existing patent application documents do not involve relevant processing techniques for spectral sparsification of scene graphs. The aforementioned existing techniques cannot retain most of the information in the graph through spectral sparsification of the graph, and the scale of the graph still needs to be reduced, and there are problems of high complexity and increased storage scale.

[0006] In summary, the existing technology has technical problems such as node redundancy in large-scale three-dimensional scene graphs, which leads to a significant increase in computing and storage scale, as well as high complexity of the scene graph. Summary of the invention

[0007] The technical problem to be solved by the present invention is: how to solve the technical problem of node redundancy in large-scale three-dimensional scene graphs in the prior art, which leads to a significant increase in calculation and storage scale, and high complexity of the scene graph.

[0008] The present invention adopts the following technical solution to solve the above technical problem: the spectral graph sparse method in the 3D scene graph includes:

[0009] S1, using sensors to collect data to collect scene three-dimensional point cloud data;

[0010] S2. constructing a scene graph, wherein a 3D scene graph generation algorithm is used to define nodes according to scene elements, edges are created according to spatial relationships between nodes, and edge weights are set;

[0011] S3. Extract node geometric features from the scene 3D point cloud data, analyze the scene graph to extract color histogram and texture features; use color histogram and texture features to enhance node description; use feature selection methods to evaluate the impact of color histogram and texture features on the graph to sort the importance;

[0012] S4, obtaining an adjacency matrix and constructing a degree matrix for the scene graph, and constructing a Laplacian matrix based on the matrix;

[0013] S5. Use the eigendecomposition algorithm to calculate the eigenvalues ​​and eigenvectors of the Laplacian matrix, and use them to identify and obtain graph structure features and potential clusters;

[0014] S6. Evaluate the contribution of the edge according to the eigenvalue and eigenvector, maintain the current spectral characteristics, identify the removable edges from the edges; perform spectral sparsification operation, retain the applicable edges, and construct a sparse graph based on them;

[0015] S7. Use graph optimization algorithms to optimize the structure of the scene graph and obtain the optimal path; verify the connectivity and validity of the sparse graph, identify isolated nodes and disconnected subgraphs, where graph optimization algorithms include: minimum spanning tree and Dijkstra algorithm.

[0016] The present invention can find a balance between the sparsity and density of the graph, and between reducing the computational and storage requirements and density. The three-dimensional scene graph is processed using graph sparse technology to reduce the number of nodes and edges in the graph data structure, while trying to maintain the original structure and important features of the graph. The memory usage and computing resource requirements are reduced, so that the system can run on a mobile robot platform with limited computing resources. At the same time, the visual or cognitive noise is removed after the sparse processing. By spectrally sparsifying the constructed three-dimensional scene graph, important nodes or edges are retained, the complexity of the scene graph is reduced, and the key parts of the scene are more prominent.

[0017] In a more specific technical solution, in S2, the spatial relationship between nodes includes: Euclidean distance, relative position and semantic relationship.

[0018] In a more specific technical solution, in S3, the node geometric features include: volume, surface area, shape descriptor, wherein the shape descriptor includes: curvature, normal information.

[0019] In a more specific technical solution, in S3, the texture features include: local binary pattern LBP and Gabor filter; the feature selection method includes: information gain and principal component analysis.

[0020] The present invention analyzes the image to extract color histogram and texture features, including local binary pattern LBP and Gabor filter. These features help to enhance the description ability of the node.

[0021] In a more specific technical solution, in S4, the matrix element A[i][j] in the adjacency matrix represents the connection relationship between node i and node j; the element D[i][i] in the degree matrix represents the degree of node i, and the degree is used to represent the number of edges connected to the current node, and the remaining elements in the degree matrix are set to 0.

[0022] In a more specific technical solution, in S4, the Laplacian matrix L of the scene graph is constructed using the following logic:

[0023] L=DA

[0024] Where A represents the adjacency matrix and D represents the construction degree matrix.

[0025] The present invention constructs a preliminary three-dimensional scene graph by the relationship between objects in the three-dimensional scene graph, extracts features from it for spectral analysis, calculates eigenvalues ​​and eigenvectors, identifies important structural features in the graph, and uses spectral sparsification technology to select and retain important nodes or edges, thereby reducing the complexity of the scene graph. Spectral analysis is performed in the preliminary constructed three-dimensional scene graph to extract and calculate features and identify important structural features in the scene graph.

[0026] In a more specific technical solution, in S5, the eigenvalue and eigenvector are calculated using the following logic:

[0027] L=UΣV T

[0028] Wherein, U and V are orthogonal matrices Σ is a diagonal matrix, and characteristic information of the Laplace matrix L is obtained according to the singular values ​​in the diagonal matrix Σ.

[0029] In a more specific technical solution, the characteristic information includes: Larulax matrix rank and information loss.

[0030] In a more specific technical solution, in S6, the spectral sparsification operation includes: edge sampling, setting a threshold, and retaining important edges.

[0031] The present invention evaluates the contribution of edges based on eigenvalues ​​and eigenvectors, and identifies edges that can be removed while maintaining spectral characteristics. Edge sampling or other selection strategies are used to retain important edges and construct a sparse graph. This step can determine which edges are retained by setting a threshold, ensuring that the graph is simplified while retaining important information.

[0032] In order to solve the problem of node redundancy in three-dimensional scene graphs and the significant increase in computing and storage scale as the scale increases, the present invention uses graph sparse technology to process the three-dimensional scene graph, reduce the number of nodes and edges in the graph data structure, and try to maintain the original structure and important features of the graph.

[0033] In a more specific technical solution, the spectral graph sparsification system in the 3D scene graph includes:

[0034] A three-dimensional point cloud acquisition module is used to collect three-dimensional point cloud data of the scene using sensors to collect data;

[0035] A scene graph construction module is used to construct a scene graph, wherein a 3D scene graph generation algorithm is used to define nodes according to scene elements, edges are created according to the spatial relationship between nodes, and edge weights are set. The scene graph construction module is connected to the 3D point cloud acquisition module;

[0036] The feature analysis and extraction module is used to extract node geometric features from the scene 3D point cloud data, analyze the scene graph to extract color histogram and texture features; use the color histogram and texture features to enhance the node description; use the feature selection method to evaluate the impact of the color histogram and texture features on the graph to sort the importance. The feature analysis and extraction module is connected with the scene graph construction module and the 3D point cloud acquisition module;

[0037] A Laplace matrix construction module is used to obtain an adjacency matrix and construct a degree matrix for a scene graph, and to construct a Laplace matrix. The Laplace matrix construction module is connected to the scene graph construction module.

[0038] A feature clustering recognition module is used to calculate the eigenvalues ​​and eigenvectors of the Laplace matrix using the eigendecomposition algorithm, so as to identify and obtain graph structure features and potential clusters. The feature clustering recognition module is connected to the Laplace matrix construction module.

[0039] The spectral sparsification module is used to evaluate the contribution of the edge according to the eigenvalue and eigenvector, maintain the current spectral characteristics, and identify the removable edges from the edges; perform spectral sparsification operations, retain applicable edges, and construct a sparse graph based on them. The spectral sparsification module is connected to the feature clustering recognition module;

[0040] The scene graph optimization module is used to optimize the structure of the scene graph and obtain the optimal path by using the graph optimization algorithm; verify the connectivity and validity of the sparse graph, and identify isolated nodes and disconnected subgraphs. The graph optimization algorithms include: minimum spanning tree and Dijkstra algorithm. The scene graph optimization module is connected to the spectral sparsification module.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention can find a balance between the sparsity and density of the graph, and between reducing the computational and storage requirements and density. The three-dimensional scene graph is processed using graph sparse technology to reduce the number of nodes and edges in the graph data structure, while trying to maintain the original structure and important features of the graph. The memory usage and computing resource requirements are reduced, so that the system can run on a mobile robot platform with limited computing resources. At the same time, the visual or cognitive noise is removed after the sparse processing. By spectrally sparsifying the constructed three-dimensional scene graph, important nodes or edges are retained, the complexity of the scene graph is reduced, and the key parts of the scene are more prominent.

[0043] The present invention analyzes the image to extract color histogram and texture features, including local binary pattern LBP and Gabor filter. These features help to enhance the description ability of the node.

[0044] The present invention constructs a preliminary three-dimensional scene graph by the relationship between objects in the three-dimensional scene graph, extracts features from it for spectral analysis, calculates eigenvalues ​​and eigenvectors, identifies important structural features in the graph, and uses spectral sparsification technology to select and retain important nodes or edges, thereby reducing the complexity of the scene graph. Spectral analysis is performed in the preliminary constructed three-dimensional scene graph to extract and calculate features and identify important structural features in the scene graph.

[0045] The present invention evaluates the contribution of edges based on eigenvalues ​​and eigenvectors, and identifies edges that can be removed while maintaining spectral characteristics. Edge sampling or other selection strategies are used to retain important edges and construct a sparse graph. This step can determine which edges are retained by setting a threshold, ensuring that the graph is simplified while retaining important information.

[0046] In order to solve the problem of node redundancy in three-dimensional scene graphs and the significant increase in computing and storage scale as the scale increases, the present invention uses graph sparse technology to process the three-dimensional scene graph, reduce the number of nodes and edges in the graph data structure, and try to maintain the original structure and important features of the graph.

[0047] The present invention solves the technical problems existing in the prior art of node redundancy in large-scale three-dimensional scene graphs, which leads to a significant increase in the scale of calculation and storage, as well as high complexity of the scene graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of basic steps of a method for spectral graph sparsification in a 3D scene graph according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example 1

[0051] like Figure 1 As shown, the spectral graph sparseness method in the 3D scene graph provided by the present invention includes the following basic steps:

[0052] S1, sensor data collection;

[0053] In this embodiment, suitable sensors are selected, including but not limited to: RGB-D camera, LiDAR, depth camera, to obtain accurate three-dimensional point cloud data of the scene.

[0054] S2, build scene graph;

[0055] In this embodiment, a related three-dimensional scene graph generation algorithm is used to define nodes based on scene elements, including but not limited to objects, feature points or spatial regions. The related attributes of each node include but are not limited to location, type and feature description. And according to the spatial relationship between nodes, such as Euclidean distance, relative position and semantic relationship, for example, objects of the same type, edges are created. Edges can have weights to indicate the strength or importance of the relationship;

[0056] S3, feature analysis and feature extraction;

[0057] In this embodiment, the geometric features of the nodes are extracted from the three-dimensional point cloud data, including but not limited to: volume, surface area, shape descriptors. The image is analyzed to extract color histogram and texture features. These features help to enhance the descriptive ability of the nodes. The feature selection method is used to evaluate the impact of each feature on the graph, determine the importance ranking, and provide a basis for subsequent steps. In this embodiment, texture features include but are not limited to: local binary pattern LBP, Gabor filter. Feature selection methods include but are not limited to: information gain, principal component analysis; shape descriptors include but are not limited to: curvature, normal information;

[0058] S4, construct the Laplacian matrix of the scene graph;

[0059] In this embodiment, the adjacency matrix of the graph is calculated, and the matrix element A[i][j] represents the connection relationship between node i and node j, 1 represents connection, and 0 represents disconnection; the degree matrix is ​​calculated and constructed, D[i][i] is the degree of node i, indicating the number of edges connected to it, and the other elements are 0; finally, the Laplace matrix L is constructed according to the following formula:

[0060] L=DA

[0061] Compute the Laplacian matrix of the graph;

[0062] S5, calculating eigenvalues ​​and eigenvectors;

[0063] In this embodiment, a Laplacian matrix is ​​constructed according to the adjacency matrix of the scene graph, and the eigenvalues ​​and eigenvectors of the Laplacian matrix are calculated using an eigendecomposition algorithm to identify major graph structural features and potential clusters.

[0064] Specifically, according to the formula:

[0065] L=UΣV T

[0066] Where U and V are orthogonal matrices and Σ is a diagonal matrix with singular values ​​on the diagonal. The singular values ​​in Σ provide important properties about the matrix L, such as its rank and information loss;

[0067] S6, spectral sparsification;

[0068] In this embodiment, the contribution of the edge is evaluated based on the eigenvalue and eigenvector to identify the edges that can be removed while maintaining the spectral characteristics. Edge sampling or other selection strategies are used to retain important edges and construct a sparse graph. Step S6 can determine which edges are retained by setting a threshold to ensure that the graph is simplified while retaining important information;

[0069] S7. Verify the connectivity and validity of the sparse graph.

[0070] In this embodiment, graph optimization algorithms, such as minimum spanning tree and Dijkstra algorithm, are applied to optimize the structure of the scene graph, ensure the connectivity between nodes and seek the optimal path. The connectivity and validity of the sparse graph are verified to ensure that the retained structure supports subsequent tasks and to check whether there are isolated nodes or disconnected subgraphs.

[0071] In summary, the present invention can find a balance between the sparsity and density of the graph, and between reducing the computing and storage requirements and density. The three-dimensional scene graph is processed using graph sparse technology to reduce the number of nodes and edges in the graph data structure, while trying to maintain the original structure and important features of the graph. The memory usage and computing resource requirements are reduced so that the system can run on a mobile robot platform with limited computing resources. At the same time, the visual or cognitive noise is removed after the sparse processing. By spectrally sparsifying the constructed three-dimensional scene graph, important nodes or edges are retained, the complexity of the scene graph is reduced, and the key parts of the scene are made more prominent.

[0072] The present invention analyzes the image to extract color histogram and texture features, including local binary pattern LBP and Gabor filter. These features help to enhance the description ability of the node.

[0073] The present invention constructs a preliminary three-dimensional scene graph by the relationship between objects in the three-dimensional scene graph, extracts features from it for spectral analysis, calculates eigenvalues ​​and eigenvectors, identifies important structural features in the graph, and uses spectral sparsification technology to select and retain important nodes or edges, thereby reducing the complexity of the scene graph. Spectral analysis is performed in the preliminary constructed three-dimensional scene graph to extract and calculate features and identify important structural features in the scene graph.

[0074] The present invention evaluates the contribution of edges based on eigenvalues ​​and eigenvectors, and identifies edges that can be removed while maintaining spectral characteristics. Edge sampling or other selection strategies are used to retain important edges and construct a sparse graph. This step can determine which edges are retained by setting a threshold, ensuring that the graph is simplified while retaining important information.

[0075] In order to solve the problem of node redundancy in three-dimensional scene graphs and the significant increase in computing and storage scale as the scale increases, the present invention uses graph sparse technology to process the three-dimensional scene graph, reduce the number of nodes and edges in the graph data structure, and try to maintain the original structure and important features of the graph.

[0076] The present invention solves the technical problems existing in the prior art of node redundancy in large-scale three-dimensional scene graphs, which leads to a significant increase in the scale of calculation and storage, as well as high complexity of the scene graph.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for spectral graph sparsification in a 3D scene graph, characterized in that The method comprises: S1, using sensors to collect data to collect scene three-dimensional point cloud data; S2, constructing a scene graph, wherein, using a three-dimensional scene graph generation algorithm, nodes are defined according to scene elements, edges are created according to spatial relationships between the nodes, and weights of the edges are set; S3, extracting node geometric features from the three-dimensional point cloud data of the scene, analyzing the scene graph to extract color histogram and texture features; using the color histogram and texture features to enhance node description; using a feature selection method to evaluate the impact information of the color histogram and texture features on the graph to sort the importance; S4, obtaining an adjacency matrix and constructing a degree matrix for the scene graph, and constructing a Laplacian matrix accordingly; S5. Calculate the eigenvalues ​​and eigenvectors of the Laplace matrix using an eigendecomposition algorithm, and use them to identify and obtain graph structure features and potential clusters; S6, evaluating the contribution of the edge according to the eigenvalue and the eigenvector, maintaining the current spectral characteristics, identifying removable edges from the edges; performing a spectral sparsification operation, retaining applicable edges, and constructing a sparse graph based on them; S7. Optimize the structure of the scene graph using a graph optimization algorithm to obtain an optimal path; verify the connectivity and validity of the sparse graph, and identify isolated nodes and disconnected subgraphs, wherein the graph optimization algorithm includes: minimum spanning tree and Dijkstra algorithm.

2. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that In S2, the spatial relationship between the nodes includes: Euclidean distance, relative position and semantic relationship.

3. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that In S3, the node geometric features include: volume, surface area, shape descriptor, wherein the shape descriptor includes: curvature and normal information.

4. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that In S3, the texture features include: local binary pattern LBP and Gabor filter; the feature selection method includes: information gain and principal component analysis.

5. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that: In S4, the matrix element A[i][j] in the adjacency matrix represents the connection relationship between node i and node j; the element D[i][i] in the construction degree matrix represents the degree of the node i, and the degree is used to represent the number of edges connected to the current node, and the remaining elements in the construction degree matrix are set to 0.

6. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that: In S4, the Laplacian matrix L of the scene graph is constructed using the following logic: L=DA Where A represents the adjacency matrix and D represents the construction degree matrix.

7. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that: In S5, the eigenvalue and the eigenvector are calculated using the following logic: L=UΣV T Wherein, U and V are orthogonal matrices and Σ is a diagonal matrix, and characteristic information of the Laplace matrix L is obtained according to the singular values ​​in the diagonal matrix Σ.

8. The method for spectral graph sparsification in a 3D scene graph according to claim 7, characterized in that: The characteristic information includes: Larulax matrix rank and information loss.

9. The method for spectral graph sparsification in a 3D scene graph according to claim 1, characterized in that: In S6, the spectral thinning operation includes: edge sampling, setting a threshold, and retaining important edges.

10. A spectral graph sparsification system in a 3D scene graph, characterized in that The system comprises: A three-dimensional point cloud acquisition module is used to collect three-dimensional point cloud data of the scene using sensors to collect data; A scene graph construction module, used to construct a scene graph, wherein a 3D scene graph generation algorithm is used to define nodes according to scene elements, edges are created according to the spatial relationship between the nodes, and weights of the edges are set. The scene graph construction module is connected to the 3D point cloud acquisition module; A feature analysis and extraction module is used to extract node geometric features from the three-dimensional point cloud data of the scene, analyze the scene graph to extract color histogram and texture features; use the color histogram and texture features to enhance node description; use a feature selection method to evaluate the impact information of the color histogram and texture features on the graph to perform importance sorting, and the feature analysis and extraction module is connected to the scene graph construction module and the three-dimensional point cloud acquisition module; A Laplace matrix construction module, used for obtaining an adjacency matrix and constructing a degree matrix for the scene graph, and constructing a Laplace matrix accordingly, wherein the Laplace matrix construction module is connected to the scene graph construction module; A feature clustering identification module, used to calculate the eigenvalues ​​and eigenvectors of the Laplace matrix using an eigendecomposition algorithm, and to identify and obtain graph structure features and potential clusters, wherein the feature clustering identification module is connected to the Laplace matrix construction module; a spectral sparsification module, used for evaluating the contribution of the edge according to the eigenvalue and the eigenvector, maintaining the current spectral characteristics, and identifying the removable edges from the edges; performing a spectral sparsification operation, retaining the applicable edges, and constructing a sparse graph accordingly, wherein the spectral sparsification module is connected to the feature clustering identification module; A scene graph optimization module, used to optimize the structure of the scene graph and obtain an optimal path by using a graph optimization algorithm; Verify the connectivity and validity of the sparse graph, identify the isolated nodes and disconnected subgraphs, wherein the graph optimization algorithm includes: minimum spanning tree, Dijkstra algorithm, and the scene graph optimization module is connected to the spectral sparsification module.

Images