3D semantic scene graph construction method based on neural network

Abstract

The invention discloses a 3D semantic scene graph construction method based on a neural network, and the method achieves the high-precision scene graph prediction through carrying out the multi-granularity analysis of a specific 3D point cloud sub-scene to extract a relation triple. Firstly, a self-adaptive instance enveloping method is designed, an approximate enveloping structure is established around an irregular instance, and shape-adaptive local point cloud sampling is provided, so that the context environment of the instance is fully covered; then, a layered dual-stage neural network is designed, and the network distinguishes and processes the features of the instances and instance pairs on different point cloud scales, so that targeted prediction is performed on instance categories and potential relationships thereof; finally, a multi-local aggregation point cloud coding module is adopted, so that local information from a plurality of point cloud sets can be combined and interacted, more comprehensive local context feature extraction is realized, and the perceptual ability of an algorithm to a point cloud scene is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data processing, and in particular relates to a method for constructing a 3D semantic scene graph based on a neural network. Background Art

[0002] Understanding complex 3D real-world environments is crucial for a variety of applications including robotics, AR / VR, and navigation. In this context, the prediction of three-dimensional semantic scene graphs (3DSSGs) has received extensive attention. The focus is on predicting instance categories in 3D point cloud environments and determining their potential relationships. For any subject-object instance pair, the relation triple <subject category, subject-object relationship, object category> structuredly represents the information of both, and is also the basic element of the 3D semantic scene graph.

[0003] Traditional methods mainly focus on global scene processing, while simultaneously identifying all instances and their potential relationships in these complex scenes makes it impossible to perform more refined fine-grained feature extraction. Its main limitations come from three factors:

[0004] (1) The point cloud sub-scenes obtained by the 3D bounding box sampling algorithm used are rough and uneven, resulting in incomplete contextual environment information for feature extraction;

[0005] (2) It relies on single-scale point cloud feature extraction, which cannot provide the fine-grained local patterns required for instance recognition and the large receptive field features required for instance pair relationship prediction;

[0006] (3) The PointNet algorithm used in the process of extracting local context information lacks the ability to perceive complex environments. Summary of the invention

[0007] In order to solve the shortcomings of the prior art and achieve the purpose of improving the instance classification prediction effect in 3D semantic scenes, the present invention adopts the following technical solutions:

[0008] A method for constructing a 3D semantic scene graph based on a neural network comprises the following steps:

[0009] Step 1: Obtain point cloud information, and separate the subject instance, object instance, and its context point cloud sub-scenes from the instance scene of the point cloud information;

[0010] Step 2: Construct a multi-level feature extraction backbone neural network to extract features at different levels; first extract local context features of each point in the sub-scene through multi-local aggregation point cloud encoding, then perform multi-level instance and instance pair feature aggregation on each point in the sub-scene, find and aggregate the features of the target point in multiple local point sets containing their own feature points, enhance the pattern recognition ability of each point to the surrounding neighborhood, and realize the interaction of multiple local features;

[0011] Step 3: Based on the multi-level feature extraction backbone neural network, the semantic features of instances and instance pairs are obtained and decoded to obtain instance categories and subject-object relationships;

[0012] Step 4: Return to step 1 until all sub-scenes are traversed and all relationship triplets are obtained to construct a complete 3D semantic scene graph.

[0013] Furthermore, in step 1, the subject and object instance pairs are determined, and the adaptive envelope method is used to uniformly sample the key points of the instance through the farthest point sampling method, extract the key point set in a series of instances, and use spherical sampling to capture the context point cloud information around the key points. Since the envelope structure of the object can be approximated by a finite number of spheres, the algorithm can adaptively obtain the surrounding environment of the instance; the surrounding environment of the subject and object instances is fused to generate the sub-scene required for triple feature extraction. This avoids the introduction of long-distance point clouds that are irrelevant to the target subject-object, and at the same time, retains as many continuous and important contextual environment point clouds as possible around the instance.

[0014] Furthermore, the sub-scene generation formula is as follows:

[0015]

[0016] in, denote the subject and object instances respectively, F(·) denotes the farthest point sampling algorithm, denote the main and object key points, n and n' denote the number of main and object key points, B(·) denotes the spherical sampling algorithm, R denotes the radius of spherical sampling, Represent the surrounding environment of the subject and object instances respectively, A sub-scene representing the context point cloud of a subject and object instance pair, preferably with a sampling radius of 1m.

[0017] Furthermore, in step 1, sub-scene feature addition is performed to calculate the distance between each point in the sub-scene point cloud and the subject and object instances. Further distance attribute addition is performed through the aforementioned stripped sub-scene point cloud, which can assist in the extraction of spatial relationships between instances. The distance features of each point in the sub-scene from the target instance need to be standardized to optimize the feature extraction of the neural network. The calculation formula is as follows:

[0018]

[0019] in, represents the mth point in the sub-scene S, Dist(·) represents the shortest distance from the point to the instance, and max(·) represents the maximum value operation. represents the distance feature between the mth point and the main instance, Represents the distance feature between the mth point and the object instance.

[0020] Furthermore, in step 2, for the local point set of each point in the sub-scene, position and distance encoding is introduced to improve the perception of the position of the subject and object instances;

[0021] For the center point x a The local neighborhood point set Introducing position and distance coding To represent any point x in the set i The relative geometric relationship with the center, the encoding calculation process is expressed as follows:

[0022]

[0023] Among them, p a Indicates the center point location information, p i Represents the location information of any point. Indicates the distance between the center point and the subject instance, represents the distance between any point and the subject instance, represents the distance between the center point and the object instance, represents the distance between any point and the object instance, Cat(·) represents the concatenation function, MLP POS (·) represents the feature mapping function of the encoding;

[0024] Add the original feature of the point to its corresponding code to obtain the point feature enhanced by relative spatial feature And the corresponding updated point set

[0025] Furthermore, in step 2, for the center point x a The local neighborhood point set The self-attention algorithm is used to extract local context features of each point in the sub-scene. The formula is as follows:

[0026]

[0027] Where Q, K, and V represent the query, key, and value of the self-attention mechanism, and W q , W k and W vRespectively represent the trainable weight matrices corresponding to Q, K and V, T represents the transpose of the matrix, d k represents the dimension of the key, represents the scale factor, Represents the encoded point set features.

[0028] Furthermore, the feature aggregation uses the k-nearest neighbor algorithm to search for the neighboring point set of each target point. For a local point set centered on a neighboring point, extract the point in the point set that is related to the sampling point x a Related coding features Represents the sampling point x a The relevant encoded point set features are extracted in parallel from all neighboring points to construct a sampling point feature set from a multi-neighborhood perspective. After obtaining the feature set, the encoding uses a set of shared parameter MLP (Shared MLP) to aggregate features in the feature channel dimension, and finally obtains the aggregated feature g a After the feature aggregation, the basic geometric attributes belonging to the instance and instance pair are also attached to the aggregated features in the form of feature encoding. The formula is as follows:

[0029]

[0030] Where S represents the sub-scenario, kNN(·) represents the k-nearest neighbor algorithm function, a represents the sampling point number, b and c represent the neighboring point numbers of the sampling point, and x k represents the kth neighboring point, and Ga(·) represents the feature aggregation function at the channel level.

[0031] Furthermore, in the step 1, sub-scene feature addition is performed, and a subject-object instance mask is added to the stripped sub-scene to guide instance identification in the relation triplet; in the step 3, in two specific stages with different resolutions in the backbone neural network, the features of the subject, object instance and the instance pair of the two in the sub-scene are stripped and aggregated respectively through the subject-object instance mask, the features of the subject and object instance are the aggregated features of the feature maps corresponding to their instance masks, and the features of the subject-object instance pair are the aggregated features of the union of the feature maps of the two.

[0032] Furthermore, in step 1, the point cloud semantic segmentation algorithm is used to segment the instances of each category in the original point cloud, and the point cloud clustering algorithm is used to cluster the single instance to obtain the mask of each instance in the environment. The formula is as follows:

[0033]

[0034] in, Represents each instance in the environment. For each instance, a unique instance mask {m1,...,m n}, Cluster(·) represents the clustering algorithm, and Seg(·) represents the indoor point cloud semantic segmentation algorithm.

[0035] Furthermore, in step 3, feature encoding for additionally representing basic geometric properties of objects is performed for single instances and pairs of instances, thereby reducing the loss of basic geometric properties caused by downsampling of the backbone network;

[0036] The basic geometric features added to a single instance include the three-dimensional coordinates of the geometric center of the object g = (g x ,g y ,g z ) and the size of the object bounding box s = (s x ,s y ,s z ), subscripts x, y, and z represent the three-dimensional components respectively;

[0037] The basic geometric features added to the instance pair include the difference in the geometric center of the two objects, the difference in the size of the bounding box, and the object volume v=s x s y s z The difference between

[0038] For the two types of feature supplementation, two groups of multi-layer perceptrons are used for encoding respectively, and the algorithm formula is as follows:

[0039]

[0040] δ rel =MLP rel (Cat(g i -g j ,s i -s j ,ln(v i / v j )))

[0041] Among them, MLP ins (·) and MLP rel (·) represent instance encoding function and relation encoding function respectively, Cat(·) represents feature concatenation function, represents the encoded i-th instance feature, δ rel Representing the encoded strength pair features, preferably, the difference in volume between objects is processed using a logarithmic function to reduce the value range and stabilize network training;

[0042] Then, two groups of multi-layer perceptrons (MLPs) are used to decode the instance features and instance pair features to obtain predicted relationship triplets. The model uses a weighted combination loss as the total loss. Preferably, for the prediction of instance categories, cross-entropy is used as the loss function, and for the prediction of instance pair relationships, binary-cross-entropy is used as the loss function.

[0043] The advantages and beneficial effects of the present invention are:

[0044] The present invention is a method for constructing 3D semantic scene graphs based on neural networks. Through detailed and comprehensive sub-scene construction, multi-scale targeted point cloud feature extraction network and encoder integrating multiple local point set features, high-precision scene graph construction is achieved, and a new paradigm for 3D scene graph prediction is established. Experiments show that the present invention achieves the best effect of 3D semantic scene graph prediction at present, especially improving the instance classification effect in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a method for constructing a 3D semantic scene graph based on a neural network in an embodiment of the present invention.

[0046] Figure 2 This is an overview of the process of constructing a 3D semantic scene graph based on a neural network in an embodiment of the present invention.

[0047] Figure 3 A schematic diagram of an adaptive instance envelope constructed in an embodiment of the present invention.

[0048] Figure 4 A schematic diagram of a hierarchical feature extraction backbone neural network constructed in an embodiment of the present invention.

[0049] Figure 5 A schematic diagram of a multi-local aggregation point cloud encoding module constructed in an embodiment of the present invention.

[0050] Figure 6a This is one of the renderings of the 3D semantic scene based on the neural network constructed in the embodiment of the present invention.

[0051] Figure 6b This is the second rendering of the 3D semantic scene based on the neural network constructed in an embodiment of the present invention.

[0052] Figure 6c This is the third rendering of the 3D semantic scene based on the neural network constructed in the embodiment of the present invention.

[0053] Figure 6d This is the fourth rendering of the 3D semantic scene based on the neural network constructed in the embodiment of the present invention. DETAILED DESCRIPTION

[0054] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0055] like Figure 1 , Figure 2 As shown, a 3D semantic scene graph construction method based on a neural network is proposed. In order to verify the effectiveness of the method, the present invention uses a public large-scale 3DSSG dataset, which includes the following steps:

[0056] Step 1: Sub-scene stripping: A general and effective adaptive instance envelope method is used on the input original point cloud information to strip the sub-scene containing the target instance pair from the original scene and retain the contextual environment information around the instance as much as possible.

[0057] For each complex scene in the dataset, we first select the instance pairs to be processed and apply the adaptive instance envelope method to separate them from the overall scene. For each sub-scene, the number of sampled points can fluctuate according to the complexity of the object, but it is preferably in the range of [2048,8192].

[0058] like Figure 3 As shown, the adaptive instance envelope method should avoid introducing distant point clouds that are irrelevant to the target subject-object, while retaining as many continuous and important contextual point clouds as possible around the instance. Preferably, in the indoor point cloud, the farthest point sampling method is used to uniformly sample the key points of the instance, and spherical sampling is performed on all key points to construct a subscene that adapts to the shape of the instance. Preferably, the radius of the spherical sampling is 1m.

[0059] Specifically, the purpose of the adaptive instance envelope method is to extract the original complete point cloud scene Strip out the instance containing the main body Object instance Subscene of its context point cloud The details are as follows:

[0060] For the main instance The algorithm first uses the farthest point sampling algorithm to extract a set of key points in a series of instances The sampling points are evenly distributed in the irregular instances. After obtaining the key points, the algorithm uses spherical sampling to capture the context point cloud around the instance. Since the envelope structure of the object can be approximated by a finite number of spheres, the algorithm can adaptively obtain the surrounding environment of the instance. After fusing the surrounding scenes of the two instances, the algorithm finally generates the sub-scene required for the triple feature extraction. The mathematical expression of the algorithm is as follows:

[0061]

[0062] Wherein, B(·) and F(·) represent the spherical sampling algorithm and the farthest point sampling algorithm respectively, and R represents the radius of the spherical sampling. Preferably, the sampling radius is 1m.

[0063] Furthermore, sub-scene feature addition. Add subject-object instance masks to the aforementioned stripped sub-scenes to guide instance identification in the relation triples, and calculate the distance between each point in the sub-scene point cloud and the target instance. Further distance and other attributes are added through the aforementioned stripped sub-scene point cloud to assist in the extraction of spatial relationships between instances. The distance features of each point in the sub-scene from the target instance need to be standardized to optimize the feature extraction of the neural network.

[0064] For input scenes First, we need to use the point cloud semantic segmentation algorithm to segment the instances of each category in the original indoor point cloud, and further use the point cloud clustering algorithm to cluster the single instance to obtain the mask of each instance in the environment. The formula is as follows:

[0065]

[0066] in, Represents each instance in the environment. For each instance, a unique instance mask {m1,...,m n}, Cluster(·) represents a clustering algorithm. Preferably, the clustering algorithm uses DBSCAN. Seg(·) represents an indoor point cloud semantic segmentation algorithm. Optionally, the algorithm uses Point Transformer.

[0067] The following is a schematic diagram of the additional properties in the sub-scene: Figure 3 As shown in the three sub-figures on the right. It is necessary to calculate the distance features from this point to the nearest point of the subject and object respectively. The calculation formula is as follows:

[0068]

[0069] Where Dist(·) represents the calculation of the shortest distance from a point to a set.

[0070] Step 2: Backbone neural network feature extraction. Based on the sub-scene point cloud extracted above as input, a set of hierarchical feature extraction backbone neural networks are used to extract features of the point cloud at different resolutions, such as Figure 4 shown.

[0071] In an embodiment of the present invention, the backbone neural network includes 6 scales, and each scale downsamples the original sub-scene point cloud to different degrees to obtain point cloud features with different resolutions. The point cloud downsampling algorithm used is the farthest point sampling algorithm. Specifically, the downsampling rates of each scale in the backbone are [1, 4, 4, 2, 2, 2].

[0072] Furthermore, multi-local aggregated point cloud encoding. In the feature extraction backbone neural network, the multi-local aggregated point cloud encoding module uses the self-attention method to extract local context features of each point. For the local point set of each point in the sub-scene, the module introduces position and distance encoding to improve the perception of the position of the subject and object instances.

[0073] Specifically, a multi-local aggregation point cloud encoding module is used to perform local feature aggregation on point cloud features of different resolutions to enhance the pattern recognition ability of each point to the surrounding neighborhood. The encoding module also adds position and distance encoding to improve the perception of the subject and object positions, such as Figure 5 shown.

[0074] For the center point x a The local neighborhood point set Introducing position and distance coding To represent any point x in the set i The relative geometric relationship with the center. The encoding calculation process is expressed as follows:

[0075]

[0076] Among them, MLP POS (·) represents the feature mapping function of the code. Adding the original feature of the point to its corresponding code can obtain the point feature enhanced by relative spatial features. And the corresponding updated point set

[0077] For the center point x a The local neighborhood point set The self-attention algorithm is used to extract features from the set. Its mathematical representation is as follows:

[0078]

[0079] Among them, W q , W k and W v Represent the trainable weight matrices, represents the encoded point set features, is the scale factor.

[0080] Furthermore, triple feature aggregation and basic geometric attribute addition are performed. For each point in the sub-scene, the multi-local aggregation point cloud encoding module further searches for and aggregates the features of the target point in multiple local point sets containing its own feature points, thus realizing the interaction of multiple local features.

[0081] After performing complete sub-scene feature extraction, the present invention performs feature aggregation of instances and instance pairs at the 4th and 6th scales of the backbone network. After feature aggregation, the basic geometric attributes belonging to the instances and instance pairs are also attached to the aggregated features in the form of feature encoding. Figure 4 The middle part is shown.

[0082] The multi-local aggregation point cloud encoding module realizes the feature interaction and aggregation in different local point sets, and uses the k-nearest neighbor algorithm to search the surrounding local neighborhood containing its features for each target point; preferably, the number of k-nearest neighbors searched is 8.

[0083] Specifically, for the sampling point x a The module first uses k-neighborhood sampling to search its neighboring point set For a local point set centered on a neighboring point, the module extracts the point in the point set that is related to the sampling point x a Related coding features By extracting relevant features from all neighboring points in parallel, the module can construct a feature set of sampling points from a multi-neighborhood perspective. Among them, s represents the number of local neighborhoods involved, and c represents the dimension of the encoded feature. After obtaining the feature set, the encoding uses a set of shared parameter MLP (Shared MLP) to perform feature aggregation in the feature channel dimension, and finally obtains the feature g a The mathematical expression of this process is as follows:

[0084]

[0085] Where Ga(·) represents the feature aggregation function at the channel level.

[0086] Step 3: Triple feature decoding. In the above backbone neural network, there are two specific stages with different resolutions. The features of the subject, object instance, and the pair of the subject and object instance in the sub-scene are respectively stripped and aggregated through the subject-object instance mask. After obtaining the semantic features of the instance and the instance pair, the network uses two sets of decoders to decode the acquired instance features, and finally outputs the category of the instance and the relationship between the subject and the object. Figure 4 As shown on the right, the instances and the relationships between them are classified, and the relationship triplets contained in the sub-scene are predicted.

[0087] Specifically, the features of the subject and object instances are the aggregated features of the feature maps corresponding to their instance masks, and the features of the subject-object instance pair are the aggregated features of the union of the two feature maps. Preferably, the feature aggregation method is maximum pooling. Preferably, the feature aggregation at the instance level is located at scale 4 of the backbone network, and the feature aggregation at the instance pair relationship level is located at scale 6 of the backbone network. Scales 1 to 4 can be regarded as the instance feature extraction stage, and scales 5 to 6 can be regarded as the instance pair relationship feature extraction stage. Additional feature encoding that characterizes the basic geometric properties of objects is provided for single instances and instance pairs to reduce the loss of basic geometric properties caused by downsampling of the backbone network.

[0088] Specifically, the basic geometric features supplemented for a single instance include the three-dimensional coordinates of the geometric center of the object g = (g x ,g y ,g z ), the size of the object bounding box s = (s x ,s y ,s z ). The basic geometric features added to the example pair are the difference in geometric centers of the two objects, the difference in the enclosing dimensions, and the volume v = s of the object. x s y s z For the two types of feature supplementation, the algorithm uses two sets of multi-layer perceptrons for encoding. Its mathematical expression is as follows:

[0089]

[0090] δ rel =MLP rel (Cat(g i -g j ,s i -s j ,ln(v i / v j )))

[0091] Among them, MLP ins (·) and MLP rel (·) respectively represent encoding functions, and Cat(·) represents the feature concatenation process. Preferably, the volume difference between objects is processed using a logarithmic function to reduce the value range and stabilize network training.

[0092] The network uses two sets of MLP to decode instance and instance pair features to obtain predictions of relation triplets. The model uses a weighted combined loss as the total loss. Preferably, for instance category prediction, cross-entropy is used as the loss function, and for instance pair relationship prediction, binary-cross-entropy is used as the loss function.

[0093] Step 4: Traverse the global sub-scene. Traverse all possible relationship triplets in the scene to build a complete 3D semantic scene graph.

[0094] For a scene with multiple instances, fully constructing a 3D semantic scene graph requires pairing the instances in pairs, performing feature predictions on the relationship triples for each group of instances, and finally traversing to obtain the complete scene graph.

[0095] Specifically, for a scene containing 10 instances, the number of instance combinations that need to be traversed is In the 3DSSG dataset, Figures 6a to 6d shown.

[0096] Experimental results show that this method demonstrates outstanding performance in large-scale 3D semantic scene graph prediction datasets and achieves better results than traditional models.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a 3D semantic scene graph based on a neural network, characterized in that The steps include: Step 1: Obtain point cloud information, and separate the subject instance, object instance, and its context point cloud sub-scenes from the instance scene of the point cloud information; Step 2: Construct a multi-level feature extraction backbone neural network to extract features at different levels; first extract local context features of each point in the sub-scene through multi-local aggregation point cloud encoding, then perform multi-level instance and instance pair feature aggregation on each point in the sub-scene, and find and aggregate the features of the target point in multiple local point sets that contain their own feature points; Step 3: Based on the multi-level feature extraction backbone neural network, the semantic features of instances and instance pairs are obtained and decoded to obtain instance categories and subject-object relationships; Step 4: Return to step 1 until all sub-scenes are traversed and all relationship triplets are obtained to construct a complete 3D semantic scene graph.

2. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: In the step 1, a pair of subject and object instances is determined, an adaptive envelope method is used, key points of the instance are uniformly sampled by the farthest point sampling method, a set of key points in a series of instances is extracted, and spherical sampling is used to capture contextual point cloud information around the key points to obtain the surrounding environment of the instance; the surrounding environment of the subject and object instances is fused to generate a sub-scene.

3. The method for constructing a 3D semantic scene graph based on a neural network according to claim 2, characterized in that: The sub-scene generation formula is as follows: in, denote the subject and object instances respectively, F(·) denotes the farthest point sampling algorithm, denote the main and object key points, n and n' denote the number of main and object key points, B(·) denotes the spherical sampling algorithm, R denotes the radius of spherical sampling, Represent the surrounding environment of the subject and object instances respectively, A subscene representing the context point cloud of a subject and object instance pair.

4. The method for constructing a 3D semantic scene graph based on a neural network according to claim 3, characterized in that: In step 1, sub-scene features are added to calculate the distance between each point in the sub-scene point cloud and the subject and object instances. The calculation formula is as follows: in, represents the mth point in the sub-scene S, Dist(·) represents the shortest distance from the point to the instance, and max(·) represents the maximum value operation. represents the distance feature between the mth point and the main instance, Represents the distance feature between the mth point and the object instance.

5. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: In step 2, for the local point set of each point in the sub-scene, position and distance coding is introduced; For the center point x a The local neighborhood point set Introducing position and distance coding To represent any point x in the set i The relative geometric relationship with the center, the encoding calculation process is expressed as follows: Among them, p a Indicates the center point location information, p i Represents the location information of any point. Indicates the distance between the center point and the subject instance, represents the distance between any point and the subject instance, represents the distance between the center point and the object instance, represents the distance between any point and the object instance, Cat(·) represents the concatenation function, MLP POS (·) represents the feature mapping function of the encoding; Add the original feature of the point to its corresponding code to obtain the point feature enhanced by relative spatial feature And the corresponding updated point set 6. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: In step 2, for the center point x a The local neighborhood point set The self-attention algorithm is used to extract local context features of each point in the sub-scene. The formula is as follows: Among them, Q, K and V represent the query, key and value of the self-attention mechanism, and W q , W k and W v Respectively represent the trainable weight matrices corresponding to Q, K and V, T represents the transpose of the matrix, d k represents the dimension of the key, represents the scale factor, Represents the encoded point set features.

7. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: The feature aggregation uses the k-nearest neighbor algorithm to search for each target point and its neighboring point set For a local point set centered on a neighboring point, extract the point in the point set that is related to the sampling point x a Related coding features Represents the sampling point x a The relevant encoded point set features are extracted in parallel from all neighboring points to construct a sampling point feature set from a multi-neighborhood perspective. And perform feature aggregation in the feature channel dimension, and finally obtain the aggregated feature g a , the formula is as follows: Where S represents the sub-scenario, kNN(·) represents the k-nearest neighbor algorithm function, a represents the sampling point number, b and c represent the neighboring point numbers of the sampling point, and x k represents the kth neighboring point, and Ga(·) represents the feature aggregation function at the channel level.

8. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: In the step 1, sub-scene features are added, and a subject-object instance mask is added to the stripped sub-scene; in the step 3, in the backbone neural network, the features of the subject, object instance, and the instance pair of the two in the sub-scene are stripped and aggregated respectively through the subject-object instance mask, the features of the subject and object instance are the aggregated features of the feature maps corresponding to their instance masks, and the features of the subject-object instance pair are the aggregated features of the union of the feature maps of the two.

9. The method for constructing a 3D semantic scene graph based on a neural network according to claim 8, characterized in that: In step 1, the point cloud semantic segmentation algorithm is used to segment the instances of each category in the original point cloud, and the point cloud clustering algorithm is used to cluster the single instance to obtain the mask of each instance in the environment. The formula is as follows: in, Represents each instance in the environment. For each instance, a unique instance mask {m1,...,m n }, Cluster(·) represents the clustering algorithm, and Seg(·) represents the indoor point cloud semantic segmentation algorithm.

10. The method for constructing a 3D semantic scene graph based on a neural network according to claim 1, characterized in that: In the step 3, feature encodings representing basic geometric properties of objects are added to individual instances and instance pairs; The basic geometric features added to a single instance include the three-dimensional coordinates of the geometric center of the object g = (g x ,g y ,g z ) and the size of the object bounding box s = (s x ,s y ,s z ), subscripts x, y, and z represent the three-dimensional components respectively; The basic geometric features added to the instance pair include the difference in the geometric center of the two objects, the difference in the size of the bounding box, and the object volume v=s x s y s z The difference between For the two types of feature supplementation, two groups of multi-layer perceptrons are used for encoding respectively, and the algorithm formula is as follows: δ rel =MLP rel (Cat(g i -g j ,s i -s j ,ln(v i / v j ))) Among them, MLP ins (·) and MLP rel (·) represent instance encoding function and relation encoding function respectively, Cat(·) represents feature concatenation function, represents the encoded i-th instance feature, δ rel represents the encoded strength pair characteristics; Then, two groups of multi-layer perceptrons are used to decode the instance features and instance pair features to obtain the predicted relationship triples.

Images