A zero-shot indoor object detection method based on graph convolutional neural network

Abstract

This invention discloses a zero-shot indoor object detection method based on a graph convolutional neural network. This method is primarily applicable to indoor object detection projects in the absence or low-sample conditions. The algorithm designs a U-shaped graph convolutional neural network and implements a graph network downsampling layer based on center-neighborhood similarity. The algorithm addresses the lack of target samples in indoor object detection, reduces the need for manual labeling, expands the detection range, and improves detection efficiency.

Description

Technical Field

[0001] The present invention is a zero-sample detection algorithm for indoor objects based on graph convolutional neural network technology, which relates to an indoor object detection method based on deep learning technology and belongs to the field of image technology. Its detection model is based on an improved graph convolutional neural network model. Background Art

[0002] In the field of indoor object detection, deep learning-based target detection methods have achieved remarkable success on well-annotated training data. Methods such as Faster-RCNN and YOLO achieve high-performance target detection by training detectors to accurately predict target bounding boxes to match the ground truth bounding boxes while suppressing bounding boxes that are part of the background image.

[0003] However, these methods rely on large amounts of labeled training data and are unable to detect unlabeled object types. In reality, in industrial settings, object categories often have a long tail distribution, with many types of objects often lacking labeled data. Furthermore, objects of the same type often have complex appearances and numerous variations. Therefore, manual labeling is not only time-consuming and labor-intensive, but also fails to generate sufficient sample data for certain types. Based on these considerations, we propose a zero-shot learning-based indoor object detection algorithm for industrial applications. Summary of the Invention

[0004] In order to overcome the problem that researchers need to manually label a large amount of training data when a large number of samples need to be labeled to train deep learning algorithms, the present invention proposes a solution to this problem, which can automatically identify objects of unlabeled categories without the need for labeling, thereby achieving the purpose of fast and efficient detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for indoor object classification based on zero-shot learning, the method comprising the following steps:

[0006] Step 1: Filter images related to indoor object categories from large open source datasets to create an indoor object dataset as an unseen class in the training process;

[0007] Step 2: Delete the categories related to indoor objects in the ImageNet dataset to obtain the ImageNetD dataset, which is used as the visible class in the training process; obtain the word vectors of each category in the ImageNetD dataset and the word vectors of each category in the indoor object dataset;

[0008] Step 3: Based on the word vectors obtained in step 2 and the relationships between categories defined by experts, construct a knowledge graph G0 that includes the ImageNetD and indoor object dataset categories;

[0009] Step 4: Use the pre-trained convolutional neural network to fine-tune the ImageNetD dataset in step 1 and extract the weights of the last layer classifier.

[0010] Step 5: Build the ZS-GraphUNet graph convolutional neural network;

[0011] Step 6: Take the indoor object knowledge graph G0 in step 3 as input and the weight of the classifier in step 4 As a label Train the ZS-GraphUNet graph convolutional neural network;

[0012] Step 7: Input the image into the trained ZS-GraphUNet model, and the model performs forward propagation calculation;

[0013] Step 8: Output the detection results;

[0014] The specific steps of step 2 are:

[0015] Step 2-1: Create a corpus of indoor objects containing text descriptions of each object;

[0016] Step 2-2: Build a global word vector representation tool to convert the text description of each object into a word vector;

[0017] Step 2-3: Construct a co-occurrence word frequency matrix X based on the corpus. Each element x in the matrix ij Represents the number of times word i and context word j co-occur within a context window of a certain size in the corpus;

[0018] Step 2-4: Construct an approximate relationship between the word vector and the co-occurrence frequency matrix X:

[0019]

[0020] Among them, ω in the above formula i and ω j are the center and background word vectors that are finally solved; and b i and b j is the bias term of the two word vectors;

[0021] Step 2-5: Based on the AdaGrad gradient descent method, randomly sample all non-zero elements in the matrix X and minimize the loss function Get the optimal solution for the center word vector i :

[0022]

[0023] Among them, V represents the total number of categories of words in the corpus; the weight function f(X ij ) is defined as:

[0024]

[0025] The specific steps of step 4 are:

[0026] Step 4-1: Fine-tune the ResNet50 neural network model pre-trained on the ImageNet dataset on the ImageNetD dataset.

[0027] Step 4-2: Take the weight of the last layer classifier of ResNet50 after fine-tuning in 4-1 The weight is used as the true label The remaining part of ResNet50 is used as a feature extractor.

[0028] Furthermore, the specific steps of step 5 are:

[0029] Step 5-1: Build the ZS-GraphUNet network structure model, such as Figure 2 As shown in the figure, a U-shaped graph convolutional neural network is used. The basic modules of the network include the graph convolution (GCN) layer, the graph pooling (gPool) layer, and the graph upsampling (gUnpool) layer. The knowledge graph G0 is input, and a graph convolution layer is used to generate G1. G1 is downsampled by the graph pooling layer to obtain G 11 ; G 11 After a layer of graph convolution layer, we get G 21 , G 21 Downsampling to get G 22 , G 22 After a layer of graph convolution, we get G 31 ; G 31 After the graph upsampling layer, we get G 32 , and restore to G 21 The shape of the two corresponding nodes is added to obtain the fused feature G 42 ; Same, G 31 Perform feature fusion between G1 and G 43 ; G 43 After a layer of graph convolution layer, the target classifier is obtained

[0030] Furthermore, the specific steps of step 6 are:

[0031] Step 6-1: Take the knowledge graph G0 in step 3 as input and output the target classifier The classifier weights obtained in step 4 As the true label

[0032] Step 6-2: Output Perform L2 regularization and adjust the output to the corresponding range;

[0033] Step 6-4: The loss function uses Leaky-ReLu loss with mask to minimize the target classifier Visible class classifier weights and true labels the gap;

[0034] Step 6-3: Use the Adam method to iterate during training to complete the training of the ZS-GraphUNet network;

[0035] Furthermore, the specific steps of step 8 are:

[0036] Step 8-1: Input the image of the indoor object dataset into the feature extractor in step 4 for forward calculation and output the visual features of the image;

[0037] Step 8-2: Input the image features extracted in step 8-1 into the trained target classifier After a softmax, the probability distribution of the image in each category is obtained;

[0038] Step 8-3: The category with the largest output probability distribution is the result of zero-sample classification.

[0039] The present invention designs a U-shaped graph convolutional neural network and implements a graph network downsampling layer based on center-neighborhood similarity. The algorithm involved in the invention solves the problem of lack of target samples in the field of indoor object detection, reduces the need for manual labeling, expands the detection range, and improves detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the original image in the present invention.

[0041] Figure 2 It is a schematic diagram of the framework of the model in the present invention.

[0042] Figure 3 Schematic diagram of the network of the downsampling layer in the present invention.

[0043] Figure 4 It is a result schematic diagram of the present invention. DETAILED DESCRIPTION

[0044] The zero-sample indoor object detection method of the present invention is described in detail below with reference to the accompanying drawings:

[0045] Step 1: We screened images related to indoor object categories from large open-source datasets including COCO, KITTI, CityScapes, BDD100K, nuScenes, and OpenImage. We then cropped images containing indoor object categories to create an indoor object dataset. This dataset contains 782 indoor object categories and 1,187,052 images, which served as unseen classes during training.

[0046] Step 2: Delete the categories related to indoor objects in the ImageNet dataset to obtain the ImageNetD dataset, which contains 853 categories and 12,110,145 images, as the visible categories in the training process; obtain the word vectors of each category in the ImageNetD dataset and the word vectors of each category in the indoor object dataset;

[0047] The specific steps of step 2 are:

[0048] Step 2-1: Crawl text descriptions of various categories on Wikipedia to create a corpus of indoor objects, containing 1,635 text segments covering 1,635 categories;

[0049] Step 2-2: Construct a co-occurrence frequency matrix X based on the corpus. Each element x in the matrix ij Represents the number of times word i and context word j co-occur within a context window of a specific size in the corpus.

[0050] Step 2-3: Construct an approximate relationship between the word vector and the co-occurrence frequency matrix X:

[0051]

[0052] Among them, ω in the above formula i and ω j are the center and background word vectors that are finally solved; and b i and b j is the bias term between the two word vectors.

[0053] Step 2-4: Based on the AdaGrad gradient descent method, randomly sample all non-zero elements of the matrix X and minimize the loss function Get the optimal solution ω for the center word vector i , the learning rate is 0.05, and the number of training rounds is 100:

[0054]

[0055] Among them, V represents the total number of categories of words in the corpus; the weight function f(X ij ) is defined as:

[0056]

[0057] Step 3: Based on the word vectors obtained in step 2 and the relationships between categories defined by experts, construct a knowledge graph G0 that includes the ImageNetD and indoor object dataset categories;

[0058] Step 4: Use the pre-trained convolutional neural network to fine-tune the ImageNetD dataset in step 1 and extract the weights of the last layer classifier.

[0059] The specific steps of step 4 are:

[0060] Step 4-1: Fine-tune the ResNet50 neural network model pre-trained on the ImageNet dataset on the ImageNetD dataset.

[0061] Step 4-2: Take the weight of the last layer classifier of ResNet50 after fine-tuning in 4-1 The weight is used as the true label Use the remaining part of ResNet50 as a feature extractor;

[0062] Step 5: Build the ZS-GraphUNet graph convolutional neural network;

[0063] The specific steps of step 5 are:

[0064] Step 5-1: Build the ZS-GraphUNet network structure model, such as Figure 2 As shown. A U-shaped graph convolutional neural network is used. The basic modules of the network include the graph convolution (GCN) layer, the graph pooling (gPool) layer, and the graph upsampling (gUnpool) layer. The knowledge graph G0 is input, and a graph convolution layer is used to generate G1. G1 is downsampled by the graph pooling layer to obtain G 11 ; G 11 After a layer of graph convolution layer, we get G 21 , G 21 Downsampling to get G 22 , G 22 After a layer of graph convolution, we get G 31 ; G 31 After the graph upsampling layer, we get G 32 , and restore to G 21 The shape of the two corresponding nodes is added to obtain the fused feature G 42 ; Same, G 31 Perform feature fusion between G1 and G 43 ; G 43 After a layer of graph convolution layer, the target classifier is obtained

[0065] Step 6: Take the indoor object knowledge graph G0 in step 3 as input and the weight of the classifier in step 4 As a label Train the ZS-GraphUNet graph convolutional neural network;

[0066] The specific steps of step 6 are:

[0067] Step 6-1: Take the knowledge graph G0 in step 3 as input and output the target classifier The classifier weights obtained in step 4 As the true label

[0068] Step 6-2: Output Perform L2 regularization and adjust the output to the corresponding range;

[0069] Step 6-4: The loss function uses Leaky-ReLu loss with mask to minimize the target classifier Visible class classifier weights and true labels the gap;

[0070] Step 6-3: During training, the Adam method is used to iterate 300 rounds to complete the training of the ZS-GraphUNet network;

[0071] Step 7: Input the image into the trained ZS-GraphUNet model, and the model performs forward propagation calculation;

[0072] Step 8: Output the detection results;

[0073] The specific steps of step 8 are:

[0074] Step 8-1: Input the image of the indoor object dataset into the feature extractor in step 4 for forward calculation and output the visual features of the image;

[0075] Step 8-2: Input the image features extracted in step 8-1 into the trained target classifier After a softmax, the probability distribution of the image in each category is obtained;

[0076] Step 8-3: The category with the largest output probability distribution is the result of zero-sample classification.

Claims

1. A zero-shot indoor object detection method based on a graph convolutional neural network, the method comprising: Step 1: Filter images related to indoor object categories from large open source datasets to create an indoor object dataset as an unseen class in the training process; Step 2: Delete the categories related to indoor objects in the ImageNet dataset to obtain the ImageNetD dataset, which is used as the visible class in the training process; obtain the word vectors of each category in the ImageNetD dataset and the word vectors of each category in the indoor object dataset; Step 2-1: Crawl text descriptions of various categories on Wikipedia to create a corpus of indoor objects, containing 1,635 texts covering 1,635 categories. Step 2-2: Construct a co-occurrence frequency matrix X based on the corpus. Each element x in the matrix ij Represents the number of times word i and context word j co-occur within a context window of a certain size in the corpus; Step 2-3: Construct an approximate relationship between the word vector and the co-occurrence frequency matrix X: Among them, ω in the above formula i and ω j are the center and background word vectors that are finally solved; and b i and b j is the bias term of the two word vectors; Step 2-4: Based on the AdaGrad gradient descent method, randomly sample all non-zero elements of the matrix X and minimize the loss function Get the optimal solution ω for the center word vector i , the learning rate is 0.05, and the number of training rounds is 100: Among them, V represents the total number of categories of words in the corpus; the weight function f(X ij ) is defined as: Step 3: Based on the word vectors obtained in step 2 and the relationships between categories defined by experts, construct a knowledge graph G0 that includes the categories of ImageNetD and indoor object datasets; Step 4: Use the pre-trained convolutional neural network to fine-tune the ImageNetD dataset in step 1 and extract the weights of the last layer classifier. Step 5: Build the ZS-GraphUNet graph convolutional neural network; A U-shaped graph convolutional neural network is used. The basic modules of the network include graph convolution layer, graph pooling layer, and graph upsampling layer. The knowledge graph G0 is input, and G1 is generated after a graph convolution layer. G1 is downsampled by the graph pooling layer to obtain G 11 ; G 11 After a layer of graph convolution layer, we get G 21 , G 21 Downsampling to get G 22 , G 22 After a layer of graph convolution, we get G 31 ; G 31 After the graph upsampling layer, we get G 32 , and restore to G 21 The shape of the two corresponding nodes is added to obtain the fused feature G 42 ; Same, G 31 Perform feature fusion between G1 and G 43 ; G 43 After a layer of graph convolution layer, the target classifier is obtained Step 6: Take the indoor object knowledge graph G0 in step 3 as input and the weight of the classifier in step 4 As a label Train the ZS-GraphUNet graph convolutional neural network; Step 7: Input the image into the trained ZS-GraphUNet model, and the model performs forward propagation calculation; Step 8: Output the detection results.

2. The zero-shot indoor object detection method based on graph convolutional neural network according to claim 1, characterized in that: The specific steps of step 4 are: Step 4-1: Fine-tune the ResNet50 neural network model pre-trained on the ImageNet dataset on the ImageNetD dataset. Step 4-2: Take the weight of the last layer classifier of ResNet50 after fine-tuning in 4-1 The weight is used as the true label The remaining part of ResNet50 is used as a feature extractor.

3. The zero-shot indoor object detection method based on graph convolutional neural network according to claim 1, characterized in that: The specific steps of step 6 are: Step 6-1: Take the knowledge graph G0 in step 3 as input and output the target classifier The classifier weights obtained in step 4 As the true label Step 6-2: Output Perform L2 regularization and adjust the output to the corresponding range; Step 6-4: The loss function uses Leaky-ReLu loss with mask to minimize the target classifier Visible class classifier weights and true labels gap.

4. The zero-shot indoor object detection method based on graph convolutional neural network according to claim 1, characterized in that: The specific steps of step 8 are: Step 8-1: Input the image of the indoor object dataset into the feature extractor in step 4 for forward calculation and output the visual features of the image; Step 8-2: Input the image features extracted in step 8-1 into the trained target classifier After a softmax, the probability distribution of the image in each category is obtained; Step 8-3: The category with the largest output probability distribution is the result of zero-sample classification.

Images