Three-dimensional scene multi-granularity semantic segmentation method based on graph neural network model

Abstract

The invention discloses a three-dimensional scene multi-granularity semantic segmentation method based on a graph neural network model, and belongs to the technical field of computer vision image processing. Taking a three-dimensional model corresponding to the three-dimensional scene as input, and selecting different visual angles to render the three-dimensional model to obtain a plurality of two-dimensional images; segmenting the two-dimensional image obtained by rendering to obtain a two-dimensional segmentation tree containing all semantic components with different granularities under different visual angles; and constructing a three-dimensional scene multi-granularity semantic segmentation map neural network model based on the two-dimensional segmentation trees under different visual angles, and performing model optimization, thereby obtaining a three-dimensional segmentation tree containing all semantic components with different granularities. By the adoption of the method, a three-dimensional data set is not needed, the three-dimensional segmentation mask is directly obtained through an existing two-dimensional image segmentation model, and the method is efficient, feasible, low in implementation cost and high in performance.

Description

Technical Field

[0001] This invention relates to a multi-granularity semantic segmentation method based on an open set 3D model using graph neural networks, belonging to the field of computer vision image processing technology. Background Technology

[0002] Two-dimensional image segmentation techniques have emerged, including open-set image segmentation tasks such as interactive segmentation and open-vocabulary segmentation. Among these, the Segment Anything Model (SAM) (Alexander Kirillov, Eric Mintun, Nikhila Ravi et al. “Segment Anything”. In: Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV). 2023:4015–4026) can generate high-quality, multi-granularity target masks based on cues such as clicks, bounding boxes, and language input, and has demonstrated excellent zero-shot segmentation capabilities in multiple computer vision tasks. SAM is trained on large-scale two-dimensional image datasets. Due to the lack of large-scale three-dimensional datasets, it is difficult to develop models similar to SAM suitable for three-dimensional scenes. Creating suitable three-dimensional datasets requires not only three-dimensional models but also annotation. Although the advent of neural radiance fields (Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik et al. "Nerf: Representing scenes as neural radiance fields for view synthesis"). In: European Conference on Computer Vision. 2020: 405–421 has enabled the reconstruction of realistic 3D scenes from images, semantic annotation of 3D scenes still requires a significant amount of manual labor and time. Summary of the Invention

[0003] To overcome the shortcomings of the existing technologies, this invention provides a multi-granular semantic segmentation method for open-set 3D scenes based on graph neural networks. This method is a 3D segmentation method based on graph neural networks and a pre-trained 2D interactive segmentation model, which can complete the open-set segmentation task of 3D models without any annotation.

[0004] This invention can be applied to semantic segmentation of 3D models represented in discrete forms such as meshes, point clouds, and Gaussian splashes. By defining the open-set 3D multi-granularity semantic segmentation task, a graph neural network model is constructed to learn a 3D segmentation tree from 2D segmentation results. This model is used to segment 3D models into all possible objects and parts of different granularities in the absence of a 3D segmentation dataset.

[0005] Specifically, this invention includes the definition of an open-set 3D multi-granularity semantic segmentation task, a 2D segmentation algorithm, and a 3D segmentation algorithm. The goal of the open-set 3D multi-granularity semantic segmentation task is to segment any semantic element of any granularity in a 2D image or 3D scene and organize them into a tree structure. The 3D segmentation algorithm first renders the 3D scene to obtain multi-view images. Then, for each image, a heuristic prompting strategy is used to obtain a set of segmentation results for each image through an existing 2D interactive segmentation model, generating a corresponding 2D segmentation tree. Finally, a graph convolutional network is used to aggregate the 2D segmentation trees from different viewpoints to obtain a 3D segmentation tree. Using this method, any 3D scene can be segmented into all objects and parts of any granularity without any annotations.

[0006] The technical solution of this invention is:

[0007] A multi-granularity semantic segmentation algorithm for 3D scenes based on a graph neural network model is proposed. For a given 3D scene and its corresponding 3D model (such as a mesh-represented model reconstructed and extracted using the NeRF algorithm), the algorithm first renders the 3D model from multiple viewpoints, obtaining several 2D images. Then, the rendered 2D images are segmented to obtain 2D segmentation trees containing all semantic objects / parts of different granularities from different viewpoints. A multi-granularity semantic segmentation graph neural network model for 3D scenes is constructed based on these 2D segmentation trees. The graph neural network model is then optimized to obtain a 3D segmentation tree containing all semantic objects / parts of different granularities. The 3D segmentation tree contains multiple nodes and edges between nodes. Each node represents a semantic object / part, and each edge connecting two nodes indicates that the semantic object / part corresponding to one node is a part of the semantic object / part corresponding to the other node.

[0008] The open-set 3D multi-granularity semantic segmentation algorithm based on graph neural networks includes the following steps:

[0009] S1 renders the 3D model corresponding to the 3D scene to obtain a 2D image, including:

[0010] S11 selects several viewpoints to see the primitives of the 3D model (i.e., the basic units that make up the 3D model, such as the triangular facets of the mesh, the points of the point cloud, etc.) to the greatest extent possible.

[0011] S12 uses the corresponding renderer (such as a rasterization-based renderer) to render the 3D model into a 2D image, and obtains the primitive number in the 3D model corresponding to each pixel in the 2D image.

[0012] S2 segments each image using a two-dimensional segmentation model algorithm (such as SAM), including:

[0013] S21 uses a pre-trained two-dimensional segmentation model to segment all possible sets of results containing objects of all granularities.

[0014] In practical implementation, two-dimensional segmentation models that can be used include SAM or its subsequent improved models HR-SAM, SematicSAM, etc.

[0015] S22 generates a two-dimensional segmentation tree based on the segmentation results of step S21; the specific process includes:

[0016] For a segmentation result s, if there is a segmentation result m in the segmentation tree that is the same as s, the two segmentation results (s and m) are directly merged; otherwise, it is determined whether there is a segmentation result m in the segmentation tree that conflicts with s. If there is, s is discarded; otherwise, the smallest segmentation result m that contains s is found, and s is inserted into the segmentation tree as a child node of m.

[0017] S3 is based on a two-dimensional segmentation tree, establishes and optimizes a graph neural network, executes a three-dimensional segmentation algorithm based on the graph neural network, and generates a three-dimensional segmentation tree;

[0018] S31 constructs an undirected weighted graph based on a two-dimensional segmentation tree and the correspondence between pixels and primitives. Where A is the weighted adjacency matrix of the undirected weighted graph, and X is the attribute matrix of each vertex of the undirected weighted graph; the undirected weighted graph is used to represent the relationship between all two-dimensional segmentation results in multiple images. Each node of the graph represents a segmentation result in the two-dimensional segmentation tree, and the edges represent the degree of overlap between two two-dimensional segmentation results.

[0019] S32 takes A and X as inputs and the results and probabilities of 3D segmentation as outputs to construct a semantic segmentation graph neural network.

[0020] The two-dimensional segmentation algorithm used can be selected from a range of algorithms, such as SAM or its subsequent improved models HR-SAM, SematicSAM, etc.; the graph neural network can include: convolutional graph neural network GCN, graph attention network GAT, etc.

[0021] The two-dimensional segmentation result is used as an attribute of the vertices of the undirected weighted graph; specifically, an autoencoder based on a fully connected layer is used to compress the two-dimensional segmentation result to obtain a feature vector of dimension D.

[0022] S33 optimized graph neural network;

[0023] S34 performs post-processing on the optimization results to obtain a three-dimensional segmentation tree.

[0024] In specific implementation, this invention selects a perspective to render a 3D model to obtain a 2D image; generates a 2D segmentation tree from the 2D image; and establishes and optimizes a graph neural network based on the 2D segmentation tree to generate a 3D segmentation tree. The representation methods of the 3D model include, but are not limited to, voxels, meshes, point clouds, and Gaussian splashing. The 2D segmentation tree includes, but is not limited to, algorithms based on interactive 2D segmentation algorithms and algorithms that directly output 2D segmentation trees; the graph neural network used includes, but is not limited to, convolutional graph neural networks (GCN), graph attention networks (GAT), etc.; the constructed graph... The methods include, but are not limited to, methods of specific implementation, such as adding the semantic features corresponding to each two-dimensional segment to the graph. The attributes of each vertex are considered. Methods for optimizing graph neural networks include, but are not limited to, specific implementation methods, such as adding new loss functions or deleting certain loss functions. The results obtained include, but are not limited to, 3D segmentation trees. Based on the 2D segmentation results, the possible 3D segmentation trees include semantic segmentation results, instance segmentation results, panoptic segmentation results, part segmentation results, etc.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] The technical solution provided by this invention enables open-set 3D multi-granularity semantic segmentation of video images based on graph neural networks. This segmentation divides the 3D scene of the video image into all objects and components of arbitrary granularity within the 3D scene. The method of this invention does not require a 3D dataset; instead, it directly obtains 3D segmentation using existing 2D image segmentation models. This method is efficient, feasible, and represents a low-cost, high-performance solution. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the process of performing multi-granular semantic segmentation of a 3D scene in an embodiment of the present invention.

[0028] Figure 2 This is a flowchart of the method for generating a two-dimensional segmentation tree in an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of the method for updating the segmentation tree in an embodiment of the present invention.

[0030] Figure 4 This is a flowchart of the method for generating a three-dimensional segmentation tree in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0032] This invention proposes a three-dimensional multi-granularity semantic segmentation method for open sets based on graph neural networks. Figure 1 The diagram shows the process of open-set 3D multi-granularity semantic segmentation based on graph neural networks provided by this invention, including: rendering a 3D model from different perspectives to obtain several 2D images; segmenting the rendered 2D images to obtain 2D segmentation trees; constructing a graph neural network based on the 2D trees from different perspectives; and optimizing the graph neural network to obtain a 3D segmentation tree.

[0033] The specific implementation includes the following steps:

[0034] S1 renders images, including:

[0035] Choose several viewpoints to see every element of the 3D model as much as possible;

[0036] For each selected viewpoint, a suitable renderer is used to render the 3D model into a 2D image, and the primitive number in the 3D model corresponding to each pixel in the image is obtained.

[0037] S2 uses a two-dimensional segmentation algorithm for each image, including:

[0038] All possible segmentation results are obtained using a pre-trained 2D segmentation model;

[0039] Generate a two-dimensional segmentation tree based on the segmentation results;

[0040] S3 performs a 3D segmentation algorithm based on graph neural networks, including:

[0041] Based on the correspondence between pixels in the corresponding image and primitives in the 3D model, an undirected weighted graph is constructed. Where A is the weighted adjacency matrix of the graph, and X is the attribute matrix of each vertex of the graph;

[0042] A graph neural network is constructed with A and X as inputs and the results and probabilities of 3D segmentation as outputs.

[0043] Optimize graph neural networks;

[0044] The optimized results are post-processed to obtain a three-dimensional segmentation tree.

[0045] The specific implementation process includes:

[0046] Image rendering:

[0047] Assume the input 3D scene's 3D model is composed of a 3D mesh. This indicates that V represents a vertex of the 3D mesh. This represents a triangular facet. In addition, V different viewpoints are given. These viewpoints need to cover as many triangular facets as possible, and each facet should be observed by at least two viewpoints. For a given viewpoint π v A rasterization renderer (such as NVDiffrast) was used to render the 3D model to obtain a 2D image. and the corresponding triangular facet index number Where H and W represent the height and width of the two-dimensional image, respectively. One pixel corresponds to one triangular facet. When Indicates image I v The pixel in row i and column j There is no corresponding triangle, meaning this pixel belongs to the background.

[0048] Two-dimensional segmentation algorithm, algorithm flow is as follows Figure 2 As shown:

[0049] Given any pre-trained interactive 2D segmentation algorithm that uses clicks as cues (such as SAM), we use a simple and direct method to segment the rendered 2D image and generate a 2D segmentation tree. First, we initialize a segmentation tree with only a root node T = {0}, where the root node represents the segmentation structure corresponding to the entire 2D image. Then, we generate regular grid-like sampling points as cues, and the interactive 2D segmentation model generates 2D segmentation results based on these cues. In a specific implementation, for example, 32x32 = 1024 points are generated, each with coordinates (4i, 4j). For each point, there are a maximum of 32x32x3 = 3072 possible segmentation results. After removing erroneous or duplicate segmentation results from these 3072 results, 100 segmentation results remain. The segmentation tree is updated using a segmentation tree update algorithm. The following process is then repeated:

[0050] Based on the current segmentation tree T, new sampling points are heuristically generated. If the number of generated points is 0, the algorithm ends. Using the generated sampling points as prompts, an interactive 2D segmentation model is used to generate 2D segmentation results. The segmentation tree is then updated using the 2D segmentation results based on the update algorithm.

[0051] Finally, after a series of post-processing steps, including compression and removal of background segmentation results, we can obtain the final two-dimensional segmentation tree T.

[0052] The process of the split tree update algorithm is as follows: Figure 3 As shown, it specifically includes:

[0053] When using two-dimensional (or three-dimensional) segmentation result set When updating the split tree T, from the set The algorithm sequentially retrieves a segmentation result s to update the segmentation tree. The process is as follows: First, let the current node m be the root node of the segmentation tree, representing the entire 3D scene. Then, repeat the following steps until the algorithm terminates:

[0054] A determines whether the current node m and the segmentation result s correspond to the same object / part, that is, whether the inclusion ratio In(s,m)>μ is satisfied. Here, μ is the "inclusion threshold", which is set to 0.9 in this embodiment.

[0055]

[0056] If s and m correspond to the same object / part, then directly merge s and m, and the algorithm ends; otherwise, traverse all child nodes q of node m:

[0057] B1 first determines whether node q contains s (whether s corresponds to the object / part or its child parts), i.e., it determines...

[0058] If In(s,q)>μ is true, then set q as the current node and repeat step A.

[0059] B2 then checks whether there is a conflict between s and q, that is, whether the intersection-union ratio (IoU(s,q)) > τ is true, where τ is the conflict threshold (set to 0.1).

[0060]

[0061] If IoU(s,q)>τ holds true, then s is an incorrect segmentation result, and the algorithm should be terminated immediately.

[0062] When C has traversed all child nodes q of node m and none of the above conditions are met, it considers s to be a child component of the current node m. First, all child components q of m that satisfy In(q,s)>μ are changed from child components of m to child components of s. Then, s is directly inserted into the split tree T, and the algorithm ends.

[0063] The process of the 3D segmentation algorithm is as follows: Figure 4 As shown, it includes:

[0064] Using the two-dimensional segmentation algorithm described above, each rendered image is segmented to generate a corresponding two-dimensional segmentation tree. The two-dimensional segmentation results yield all objects and components at arbitrary granularity. To generate a three-dimensional segmentation mask, we need to fuse the segmentation results of the same object or component from different viewpoints. We propose using graph neural networks (GNNs) to generate the three-dimensional segmentation mask and obtain the semantic segmentation categories of the three-dimensional scene.

[0065] In order to use graph neural networks, we need to construct an undirected weighted graph. This is used as the input to the graph neural network. Represents the weighted adjacency matrix of an undirected weighted graph. N = M + V represents the matrix consisting of the attributes of each vertex in an undirected weighted graph, and M = M1 + ... + M2 represents the number of vertices in the undirected weighted graph. V M is the number of all two-dimensional segmentation results. v It is the two-dimensional segmentation tree T corresponding to the v-th viewpoint. v The number of nodes in the graph, V is the number of viewpoints; D is the dimension of the eigenvectors of the vertex attributes in the undirected weighted graph. Specifically, the weighted adjacency matrix A is constructed as follows:

[0066]

[0067] in These represent the relationships between segments, between segments and viewpoints, and between viewpoints, respectively. More specifically, A mm [i,j]=IoU(s i ,s j |π i ,π j The two segmentation results were calculated simultaneously at the viewpoint π. i and π j The ratio of intersection and union of areas in the middle, A mv [i,v]=In(s i ,π v The segmentation result s was calculated. i From the perspective π v The proportion of the area in, and A vv [a,b]=In(π a ,π b The angle π was calculated. a What percentage of pixels can be viewed by the perspective π? b Observed.

[0068] We want to use the 2D segmentation results as attributes of vertices in an undirected weighted graph. However, if we directly convert the 2D segmentation results into feature vectors, the dimension N of the feature vectors will be... F The input dimension is too high and unsuitable for training graph neural networks. Therefore, we use an autoencoder based on fully connected layers to compress the 2D segmentation result into a feature vector of dimension D, which serves as the attribute of the vertices of the undirected weighted graph. The autoencoder consists of an encoder and a decoder. In this embodiment, the encoder is a fully connected neural network with 3 layers and 256 hidden neurons, using ReLU non-linear activation. The input dimension of the encoder is N. F The output dimension is D, while the input and output dimensions of the decoder are D and N, respectively.F .

[0069] Let K be a three-dimensional mesh. The maximum possible number of segmentation results (representing the upper limit of the number of possible parts after segmentation of the 3D model; in this embodiment, the value is set to 3*max(M1,…,M)). V The weight coefficient matrix is ​​generated using a graph neural network (GNN).

[0070] Z = softmax(GNN(A,X))

[0071] Here, softmax represents the normalized exponential function; therefore, we can obtain the final 3D segmentation result by weighted averaging of the 2D segmentation results.

[0072]

[0073] Where, m kf Indicates whether the k-th segmentation result of the 3D model contains the f-th element; Z ik This represents the value in the i-th row and j-th column of the weight matrix Z; ∈ = 10 -7 To avoid division by zero, σ = 0.5 represents the threshold for segmentation, s′ i It is the two-dimensional segmentation result s i The back projection, and The two-dimensional segmentation result s i Corresponding perspective π i The visibility of the lower triangular facet. In addition, we also optimized a confidence vector. use Optimize p; where p i N represents the confidence level of the i-th candidate 3D segmentation result in the final 3D segmentation tree. F m is the number of triangular faces contained in the 3D model. k This is a vector indicating whether each triangular facet is included in the k-th 3D segmentation result.

[0074] Optimized semantic segmentation graph neural network model:

[0075] The overall optimization objective of the semantic segmentation graph neural network model is to align the predicted 3D and 2D segmentation results. The overall loss function for optimization is expressed by the following formula:

[0076]

[0077] in, λ1 to λ6 are the overall loss function; λ1 to λ6 are the weighting coefficients used to control the strength of each loss function. To match the loss function; The mask loss function; Let the viewpoint loss function be used. The loss function is for a two-dimensional tree. The loss function for a 3D tree; For reconstruction loss function;

[0078] Matching loss function Consider the mapping relationship between the weight coefficient matrix Z and the 2D-3D segmentation. If a 2D segmentation result s i With a three-dimensional prediction result m k The weighting coefficient between them is Z ik If it is very high, it indicates that s i It is very likely m k A two-dimensional projection, i.e., IoU(s) i ,m k ;π i (It is) very high, and vice versa.

[0079] Mask loss function The relationship between 2D viewpoint and 3D segmentation was considered. For a given viewpoint π, a bipartite graph matching algorithm was used to obtain the optimal correspondence between 2D and 3D segmentations. Each 3D segment m... k There should be only one or no matching two-dimensional segment s k′ .therefore, Maximize IoU(s) k′ ,m k ;π), minimize the area In(m) of the 3D segmentation in this viewpoint. k (π) or reduce the confidence level p of the 3D segmentation in the final segmentation tree. k .

[0080] Viewpoint loss function It is encouraged that the set of projections of the predicted 3D segmentation from a certain viewpoint should be consistent with the set observed.

[0081] Two-dimensional tree loss function The aim is to make the structure of the predicted result obtained by projection consistent with the observed structure.

[0082] 3D tree loss function The goal is to generate a three-dimensional tree from the three-dimensional prediction results.

[0083] Reconstruction losses The goal is to reconstruct the IoU matrix using a bipartite set of two viewpoints obtained from a 3D segmentation projection.

[0084] Post-processing: After optimizing the graph neural network (GNN) and the confidence vector p, we use a 3D segmentation algorithm to obtain the 3D segmentation result by weighted averaging of the 2D segmentation results. Then, the confidence vector p is filtered out. i Segmentation results less than 0.5 (the preset number of 3D segments is M, but the actual number is less than M, so erroneous 3D segmentations are filtered out using p). Finally, the remaining segmentation results are organized into a 3D segmentation tree based on the segmentation tree update algorithm described above.

[0085] The above processing steps enable open-set 3D multi-granularity semantic segmentation of the 3D model based on graph neural networks, resulting in all objects and components of arbitrary granularity in the 3D scene.

[0086] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A multi-granularity semantic segmentation method for 3D scenes based on a graph neural network model, characterized in that, Render a 3D scene to obtain multi-view 2D images; segment the rendered 2D images to obtain 2D segmentation trees containing all semantic components of different granularities from different viewpoints; construct a 3D scene multi-granularity semantic segmentation graph neural network model based on the 2D segmentation trees from different viewpoints and optimize the model; use graph convolutional networks to aggregate the 2D segmentation trees from different viewpoints, thereby obtaining a 3D segmentation tree containing all semantic objects or components of different granularities. Specifically, the segmentation of the rendered 2D image involves using a pre-trained 2D segmentation model to segment each image, obtaining segmentation results encompassing all granularities, and then generating a 2D segmentation tree based on the segmentation results. Specifically, for a segmentation result s, if a segmentation result m exists in the segmentation tree that is identical to the s, the two segmentation results are directly merged; otherwise, it is determined whether there is a segmentation result m in the segmentation tree that conflicts with the s. If so, the segmentation result s is discarded; otherwise, the smallest segmentation result m containing the s is found, and the segmentation result s is inserted into the segmentation tree as a child node of the segmentation result m. The construction and optimization of the 3D scene multi-granularity semantic segmentation graph neural network model, i.e., the execution of 3D segmentation based on graph neural networks, includes the following process: S31 constructs an undirected weighted graph based on the correspondence between pixels in the corresponding image of the two-dimensional segmentation tree and primitives in the three-dimensional model. The undirected weighted graph includes a weighted adjacency matrix and an attribute matrix for each vertex. The two-dimensional segmentation result is used as the attribute of the vertex of the undirected weighted graph. Specifically, an autoencoder based on a fully connected layer is used to compress the two-dimensional segmentation result to obtain the feature vector of the vertex attribute of the undirected weighted graph. S32 takes an undirected weighted graph as input and constructs a semantic segmentation graph neural network model to generate a 3D segmentation mask output. S33 optimizes the semantic segmentation graph neural network model; the optimization objective is to ensure that the predicted 3D segmentation results are consistent with the 2D segmentation results. S34 performs post-processing on the optimization results to obtain a three-dimensional segmentation tree; including: The three-dimensional segmentation result is obtained by weighted averaging of the two-dimensional segmentation result using a three-dimensional segmentation algorithm; a confidence threshold is set to filter out the segmentation results with a confidence score lower than the confidence threshold; and the remaining segmentation results are then organized into a three-dimensional segmentation tree, thus realizing multi-granular semantic segmentation of three-dimensional scenes based on a graph neural network model.

2. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 1, characterized in that, The two-dimensional segmentation model used is an interactive segmentation model, including SAM, HR-SAM, or SematicSAM.

3. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 1, characterized in that, The specific process of updating the segment tree includes: Let the current node m be the root node of the split tree, representing the entire 3D scene; From the segmentation result set Take a segmentation result s one by one and determine whether the current node m and the segmentation result s correspond to the same object or part; specifically: The inclusion ratio In(s, m) is calculated using the following formula: Set a threshold μ; when the inclusion ratio In(s, m) > μ, s and m correspond to the same object or part, then directly merge s and m, and the algorithm ends; otherwise, traverse all child nodes q of node m and perform the following operations: Determine whether node q contains s, that is, determine whether s corresponds to the object, component, or sub-component of q; when node q contains s, set q as the current node and repeat the above steps. Next, determine if there is a conflict between s and q, specifically: The intersection-union ratio (IoU(s, q)) is calculated using the following formula: Set a conflict threshold τ; when the intersection-union ratio (IoU(s, q)) > T, it indicates that s is an incorrect segmentation result, and the algorithm is terminated directly. When the intersection-union ratio IoU(s,q)>τ does not hold, then s is a child component of the current node m. First, all child components q in the child components of m that satisfy In(q,s)>μ are changed from child components of m to child components of s. Then, s is directly inserted into the split tree T.

4. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 1, characterized in that, In step S31, an undirected weighted graph is constructed, denoted as: in, Represents the weighted adjacency matrix of an undirected weighted graph. N = M + V represents the matrix consisting of the attributes of each vertex in an undirected weighted graph, and M = M1 + ... + M2 represents the number of vertices in the undirected weighted graph. V M is the number of all two-dimensional segmentation results. v It is the two-dimensional segmentation tree T corresponding to the v-th viewpoint. v The number of nodes in the graph, V is the number of viewpoints; D is the dimension of the eigenvectors of the vertex attributes of the undirected weighted graph.

5. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 4, characterized in that, The weighted adjacency matrix A is constructed as follows: in These represent the relationships between segments, between segments and viewpoints, and between viewpoints, respectively.

6. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 4, characterized in that, An autoencoder based on a fully connected layer is used to compress the two-dimensional segmentation result to obtain a feature vector of dimension D, which serves as the attribute of the vertices of the undirected weighted graph.

7. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 6, characterized in that, Autoencoders include encoders and decoders based on fully connected layers.

8. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 6, characterized in that, In step S32, a semantic segmentation graph neural network model is constructed, specifically including: Generate weight coefficient matrix using graph neural network Represented as: Z = softmax(GNN(A, X)) Where max represents the number of results obtained from image segmentation; GNN stands for Graph Neural Network; The final three-dimensional segmentation result is then obtained by weighted averaging of the two-dimensional segmentation results. Represented as: Where, m kf Indicates whether the k-th segmentation result of the 3D model contains the f-th primitive; Z ik This represents the value in the i-th row and j-th column of the weight matrix Z; ∈ = 10 -7 To avoid division by zero, σ = 0.5 represents the threshold for segmentation, s′ i It is the two-dimensional segmentation result s i The back projection, and The two-dimensional segmentation result s i Corresponding perspective π i Visibility of the lower triangular facet; N F This represents the number of triangular faces contained in the 3D model.

9. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 8, characterized in that, In step S33, the overall loss function for optimizing the semantic segmentation graph neural network model is expressed as: in, λ1 to λ6 are the overall loss function; λ1 to λ6 are the weighting coefficients used to control the strength of each loss function. To match the loss function; The mask loss function; Let the viewpoint loss function be used. The loss function is for a two-dimensional tree. The loss function for a 3D tree; The reconstruction loss function.

10. The 3D scene multi-granularity semantic segmentation method based on a graph neural network model as described in claim 9, characterized in that, Optimize confidence vector The element p i This represents the confidence level of the i-th candidate 3D segmentation result in the final 3D segmentation tree.

Images