A three-dimensional point cloud scene graph generation method based on deep learning

Abstract

The application discloses a kind of three-dimensional point cloud scene graph generation methods based on deep learning, comprising the following steps: the three-dimensional point cloud data containing multiple three-dimensional objects is input to the graph convolutional neural network classification model trained, and the category of three-dimensional object and the semantic relationship between multiple three-dimensional objects are output;According to the category of three-dimensional object and the semantic relationship between multiple three-dimensional objects, three-dimensional point cloud scene graph is constructed;Wherein, the training step of the graph convolutional neural network classification model includes: obtaining training data, and the training data includes three-dimensional point cloud training data containing multiple three-dimensional objects and the 2D picture of the three-dimensional object;The graph convolutional neural network classification model is trained by the training data, and the model parameters of the graph convolutional neural network classification model are obtained;The application effectively alleviates the defect that object lacks texture information in 3D point cloud by introducing 2D picture.

Description

Technical Field

[0001] This invention relates to the field of multimodal knowledge transfer technology, and more specifically to a method for generating 3D point cloud scene graphs based on deep learning. Background Technology

[0002] Understanding the structure of 3D geometric scenes is particularly important for tasks that require interaction with the real environment. As a significant topic in this field, predicting 3D semantic scene graphs in point clouds has received renewed attention in recent years. Specifically, given a 3D point cloud and corresponding class-agnostic instances, the task aims to construct a directed graph where the nodes are objects in the scene, and the edges connecting two objects represent the relationships between them.

[0003] However, this task currently faces some challenging issues. First, pure point cloud data loses the texture details of objects, causing the network to favor predicting spatial relationships when given two objects, while ignoring semantic relationships such as "looks like" or "belongs to". In addition, due to the difficulty of scene icon annotation, the commonly used dataset 3DSSG exhibits a severe long-tail distribution, which also makes it difficult to learn some tail classes effectively.

[0004] Furthermore, existing techniques mitigate the long-tail distribution problem by pre-training a dictionary on the dataset during the training of the scene graph network. However, this method is insufficient in supplementing the texture information of the objects themselves, leading to significant problems in object recognition. In addition, due to the use of dictionary assistance, the model still relies on additional computation during testing, resulting in significant waste of space and time.

[0005] Therefore, how to provide a deep learning-based method for generating 3D point cloud scene graphs that can alleviate the lack of object textures and long-tailed distribution is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method for generating 3D point cloud scene graphs based on deep learning, which can effectively alleviate the deficiency of lack of texture information of objects in 3D point clouds, and effectively solve the problem of long-tail distribution by introducing external knowledge.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for generating 3D point cloud scene graphs based on deep learning includes the following steps:

[0009] The three-dimensional point cloud data containing multiple three-dimensional objects is input into a trained graph convolutional neural network classification model, which outputs the categories of the three-dimensional objects and the semantic relationships between the multiple three-dimensional objects.

[0010] A 3D point cloud scene map is constructed based on the category of the 3D objects and the semantic relationships between multiple 3D objects;

[0011] The training steps of the graph convolutional neural network classification model include:

[0012] Acquire training data, which includes 3D point cloud training data containing multiple 3D objects and 2D images of the 3D objects;

[0013] The pre-built graph convolutional neural network classification model is trained using the training data to obtain the model parameters of the graph convolutional neural network classification model.

[0014] Furthermore, the training steps of the graph convolutional neural network classification model are as follows:

[0015] The pre-constructed 3D backbone network model is trained using the 3D point cloud training data.

[0016] A pre-constructed 2D auxiliary training network model is trained using the 2D images; wherein the 2D auxiliary training network model acquires 3D features from the 3D backbone network model to participate in learning, and updates the model parameters of the 3D backbone network model through gradient backpropagation;

[0017] The convergence of the model is determined by comprehensively considering the loss functions of the 3D backbone network model and the 2D auxiliary training network model.

[0018] Furthermore, both the 3D backbone network model and the 2D auxiliary training network model include an encoder, a graph neural network, and a classifier;

[0019] The encoder is used to acquire node features and edge features;

[0020] The graph neural network is used to infer and update the node features and edge features to generate object features and relationship features;

[0021] The classifier is used to classify objects and relationships based on their features to obtain object categories and relationship categories.

[0022] Furthermore, the encoder includes a node encoder and an edge encoder;

[0023] The node encoder encodes 3D object features or 2D object features to obtain 3D node features or 2D node features.

[0024] The edge encoder is a multilayer perceptron, which encodes the edge initialization features and outputs the edge features; wherein, the edge initialization features include the difference between multiple sub-features between two object features.

[0025] Furthermore, based on the 3D node features and the 2D node features, the node encoder in the 3D backbone network is subjected to approximation training.

[0026] Furthermore, the 2D-assisted training network model also includes a node collaboration network and a relationship collaboration network.

[0027] In the 2D assisted training network model, the node collaboration network uses a cross-modal attention mechanism to fuse the node features in the 3D backbone network model and the 2D assisted training network model, and inputs the fused node features into the graph neural network; the relationship collaboration network uses a cross-modal main force mechanism to fuse the relationship features in the 3D backbone network model and the 2D assisted training network model, and inputs the fused relationship features into the classifier.

[0028] Furthermore, the node encoder in the 2D assisted training network model is a CLIP pre-trained visual encoder.

[0029] Furthermore, the 2D-assisted training network model also includes a triple alignment network;

[0030] The triple alignment network is used to concatenate object features and fused relation features and reduce the dimensionality through a multilayer perceptron to obtain triple visual features; it is also used to obtain triple linguistic features and align the triple visual features and the triple linguistic features.

[0031] Furthermore, the triplet language features are generated through the CLIP model, which obtains the true labels of the triples, generates natural language according to a preset language template, and encodes them through a pre-trained multimodal network model to obtain language triplet information.

[0032] The beneficial effects of this invention are:

[0033] As can be seen from the above technical solution, compared with the prior art, this invention discloses a method for generating 3D point cloud scene graphs based on deep learning. By introducing 2D images instead of simple RGB information stitching, it effectively alleviates the deficiency of lack of texture information in objects in 3D point clouds. By introducing a pre-trained multimodal network model to incorporate external knowledge, it more effectively alleviates the long-tail distribution problem in the dataset. This invention introduces knowledge during the training process, enabling the 3D backbone network to more efficiently mine effective information from simple 3D point clouds and pre-trained language models. During testing, no additional 2D images are introduced, and no additional computation time or resources are added. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 The attached figure is a schematic diagram of a method for generating a 3D point cloud scene map based on deep learning, provided by an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figure 1 This invention discloses a method for generating 3D point cloud scene graphs based on deep learning, comprising the following steps:

[0038] S1: Acquire training data, which includes 3D point cloud training data containing multiple 3D objects and 2D images of the 3D objects; wherein the positions of the objects in the point cloud training data are labeled; establish a mapping transformation matrix from 3D point cloud to 2D image based on the coordinates of the object positions, and realize the transformation of all objects in the point cloud to 2D image through the mapping transformation matrix.

[0039] S2: Train the pre-built graph convolutional neural network classification model using the training data to obtain the model parameters of the graph convolutional neural network classification model, thus completing the training of the graph convolutional neural network classification model;

[0040] In one embodiment, the training steps of the graph convolutional neural network classification model are as follows:

[0041] S21: Train a pre-built 3D backbone network model using 3D point cloud training data;

[0042] Specifically, the 3D backbone network model includes an encoder, a graph neural network, and a classifier;

[0043] The encoder comprises a node encoder and an edge encoder, used to extract 3D node features and 3D edge features, respectively. The node encoder's role: Given a class-agnostic instance mask and an input point cloud, it extracts a set of points corresponding to a semantic instance. A simple "PointNet" network is used to extract instance-level features, which are then used as the initial object features for the scene graph nodes. The edge encoder: It needs to compute the differences between several attributes between linked objects. For each instance, these attributes include the mean and standard deviation of the 3D points, the size of the bounding box, the volume, and the maximum side length. Therefore, relational features are encoded by projecting the cascaded differences of these attributes between two instances through multilayer perceptron layers.

[0044] For the extraction of 3D node features, the input 3D object features are further extracted by the node encoder to obtain node features;

[0045] For the extraction of 3D edge features, the mean and standard deviation between the features of two objects, the size of the bounding box, the volume and the difference between the maximum side length are used as the initial features of the edge. Then, a multilayer perceptron is used as an edge encoder to encode the 3D edge features.

[0046] After the graph neural network obtains the encoded 3D node features and 3D edge features, it performs information propagation and reasoning, continuously updating the 3D node features and 3D edge features to obtain the updated node features and updated edge features, which are used as 3D object features and 3D relationship features, respectively.

[0047] The classifiers include an object category classifier and a relation category classifier. The object classifier and relation classifier are used to classify the 3D object features and 3D relation features respectively, and the loss is calculated using the cross-entropy loss function.

[0048] In one embodiment, CLIP template embedding is used to initialize the weights of the object classifier in the 3D backbone network and the object category classifier in the 2D auxiliary network. The CLIP model has strong representational capabilities, so generating a sentence from the nouns of each object category using a template and encoding the features using the CLIP language model, and then using these features as the initial features for the classifier, makes the distinction between different categories more obvious and accelerates object classification training. Furthermore, prior knowledge about the objects themselves, which is contained within the language, is also included, enriching the training process.

[0049] S22: Train a pre-built 2D auxiliary training network model using 2D images; wherein, the 2D auxiliary training network model acquires 3D features from the 3D backbone network model to participate in learning, and updates the model parameters of the 3D backbone network model through gradient backpropagation;

[0050] Specifically, the 2D-assisted training network model also includes an encoder, a graph neural network, and a classifier;

[0051] The encoder includes a node encoder and an edge encoder, which are used to extract 2D node features and 2D edge features, respectively. For the extraction of 2D node features, the input 2D object features are further extracted by the node encoder to obtain node features. For the extraction of 2D edge features, the mean and standard deviation between two object features, the size of the bounding box, the difference between the volume and the maximum side length are used as the initial features of the edge. Then, a multilayer perceptron is used as the edge encoder to encode the 2D edge features.

[0052] The classifiers include object category classifiers and relation category classifiers. The object category classifiers and relation category classifiers are used to classify object features and relation features respectively, and the loss is calculated using the cross-entropy loss function.

[0053] In one embodiment, the 2D-assisted training network model further includes a node cooperation network and a relation cooperation network.

[0054] The node writing network is a cross-attention structure where the query is 2D node features, and the key and value are 3D node features. The output is object features that incorporate 3D structural information. The node collaboration network uses a cross-modal attention mechanism to fuse node features from the 3D backbone network model and the 2D auxiliary training network model, and then inputs the fused node features into the graph neural network. After obtaining the fused node features and 2D edge features, the graph neural network performs information propagation and inference to obtain updated node features and updated edge features, which serve as 2D object features and 2D relation features, respectively.

[0055] During the forward propagation of the network, the 2D-assisted training network model receives assistance from the 3D backbone network model, but the 3D backbone network model does not receive substantial assistance from the 2D-assisted training network model. However, during the gradient backpropagation phase, the node collaboration network backpropagates gradients to the 3D network, improving the 3D backbone network model. Furthermore, because this structure does not rely on 2D feature input during the forward propagation process, the 3D backbone network model performs well during the testing phase even without the participation of the 2D-assisted training network model.

[0056] The relational collaboration network structure is also a cross-attention structure. The query is a 2D relational feature, and the key and value are 3D relational features. The relational collaboration network uses a cross-modal main mechanism to fuse the relational features in the 3D backbone network model and the 2D auxiliary training network model, resulting in a 2D relational feature with the help of 3D relational features. This fused relational feature is then input into the classifier.

[0057] The classifiers include an object category classifier and a relation category classifier. The object classifier and the relation classifier are used to classify the 2D object features and the fused relation features respectively, and the loss is calculated using the cross-entropy loss function.

[0058] In one embodiment, the node encoder in the 2D-assisted training network model is a CLIP visual encoder.

[0059] In one embodiment, the 2D-assisted training network model further includes a triple alignment network.

[0060] In the triplet network, we first concatenate [object 1 features, relationship features between object 1 and object 2, object 2 features], and then use a multilayer perceptron for dimensionality reduction to obtain a triplet visual feature. In addition, we use the object category and relationship category labels of the triplet to generate a natural language sentence from our designed template "a scene of a{object 1 category}{relationship category between object 1 and object 2}a{object 2 category}", and then use CLIP's language encoder to encode it to obtain the language features of the triplet.

[0061] Since the CLIP model is a natural language-visual alignment network, we use CLIP to align the visual features and language features of the triples in order to obtain better visual features. Here we use the maximum cosine similarity as the loss function to determine the convergence of the graph neural network.

[0062] In this embodiment, the parameters of both the CLIP visual encoder and the CLIP language encoder, which encode the 2D portion, are kept constant. The object features and relational features output by the 2D-assisted training network model are supervised, and the convergence of the 2D-assisted training network model is determined using the cross-entropy loss function.

[0063] In this embodiment, the graph neural network employs a GNN structure similar to that in SGFN. It utilizes a Feature Attention (FAT) module to pass messages between nodes and edges, then obtains updated node and edge features. Each GNN module is paired with a Multi-Head Self-Attention (MHSA) module, which repeats T times to extract the final node and relation features. Thus, an object category classifier and a relation category classifier predict the elements of each possible relation triplet from the triplet features, which ultimately construct the semantic scene graph. Specifically, we can generate CLIP text embeddings, i.e., triplet linguistic features, for each real relation triplet, and regularize the corresponding triplet visual features at the end of each GNN layer in the scene graph inference module. Therefore, regularization becomes minimizing the embedding distance between the text embedding and the fused triplet features. The two inputs to regularization are the triplet visual features and the triplet linguistic features. Since the triplet linguistic features are directly encoded by a mature language model, their features have better representational power. Here we approximate the visual features of the triplet with distance, which is also to use language features to regularize the visual features and make them more representative.

[0064] S23: Use the loss function of the integrated 3D backbone network model and the multimodal auxiliary training network model to determine model convergence.

[0065] In this embodiment, the 3D node encoder and the 2D node encoder are trained to approximate each other by maximizing cosine similarity as the loss function in the approximate training process, so as to make the overall semantic space more similar and reduce the gap between 2D node features and 3D node features.

[0066] S3: Input the 3D point cloud data containing multiple 3D objects into the trained graph convolutional neural network classification model to obtain the category of the 3D objects and the semantic relationship between the multiple 3D objects;

[0067] S4: Construct a three-dimensional point cloud scene map based on the category of the three-dimensional object and the semantic relationship between multiple three-dimensional objects.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0069] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating 3D point cloud scene graphs based on deep learning, characterized in that, Includes the following steps: The three-dimensional point cloud data containing multiple three-dimensional objects is input into a trained graph convolutional neural network classification model, which outputs the categories of the three-dimensional objects and the semantic relationships between the multiple three-dimensional objects. A 3D point cloud scene map is constructed based on the category of the 3D objects and the semantic relationships between multiple 3D objects; The training steps of the graph convolutional neural network classification model include: Acquire training data, which includes 3D point cloud training data containing multiple 3D objects and 2D images of the 3D objects; The pre-constructed graph convolutional neural network classification model is trained using the training data to obtain the model parameters of the graph convolutional neural network classification model; The specific training steps of the graph convolutional neural network classification model are as follows: train a pre-constructed 3D backbone network model using the 3D point cloud training data; A pre-constructed 2D auxiliary training network model is trained using the 2D images; wherein the 2D auxiliary training network model acquires 3D features from the 3D backbone network model to participate in learning, and updates the model parameters of the 3D backbone network model through gradient backpropagation; The convergence of the model is determined by comprehensively considering the loss functions of the 3D backbone network model and the 2D auxiliary training network model. Both the 3D backbone network model and the 2D auxiliary training network model include an encoder, a graph neural network, and a classifier. The encoder is used to acquire node features and edge features; The graph neural network is used to infer and update the node features and edge features to generate object features and relationship features; The classifier is used to classify objects and relationships based on their features to obtain object categories and relationship categories.

2. The method for generating a 3D point cloud scene graph based on deep learning according to claim 1, characterized in that, The encoder includes a node encoder and an edge encoder; The node encoder encodes 3D object features or 2D object features to obtain 3D node features or 2D node features; The edge encoder is a multilayer perceptron, which encodes the edge initialization features and outputs the edge features; wherein, the edge initialization features include the difference between multiple sub-features between two object features.

3. The method for generating a 3D point cloud scene graph based on deep learning according to claim 2, characterized in that, The node encoders in the 3D backbone network are approximated and trained based on the 3D node features and the 2D node features.

4. The method for generating a 3D point cloud scene graph based on deep learning according to claim 1, characterized in that, The 2D-assisted training network model further includes a node collaboration network and a relationship collaboration network. In the 2D-assisted training network model, the node collaboration network uses a cross-modal attention mechanism to fuse the node features in the 3D backbone network model and the 2D-assisted training network model, and inputs the fused node features into the graph neural network. The relationship collaboration network uses a cross-modal attention mechanism to fuse the relationship features in the 3D backbone network model and the 2D-assisted training network model, and inputs the fused relationship features into the classifier.

5. The method for generating a 3D point cloud scene graph based on deep learning according to claim 4, characterized in that, The node encoder in the 2D assisted training network model is a CLIP pre-trained visual encoder.

6. The method for generating a 3D point cloud scene graph based on deep learning according to claim 4, characterized in that, The 2D-assisted training network model also includes a triplet alignment network; The triple alignment network is used to concatenate object features and fused relation features and reduce the dimensionality through a multilayer perceptron to obtain triple visual features; it is also used to obtain triple linguistic features and align the triple visual features and the triple linguistic features.

7. The method for generating a 3D point cloud scene graph based on deep learning according to claim 6, characterized in that, The triplet language features are generated by the CLIP model, which obtains the true labels of the triples, generates natural language according to the preset language template, and encodes them through a pre-trained multimodal network model to obtain language triplet information.

Images