A semi-supervised RGB-d object classification method based on angle prediction pre-training

Abstract

The application discloses a kind of semi-supervised RGB-D object classification methods based on angle prediction pre-training, comprising: the rotation angle predictor of RGB and depth image is trained, obtains the network model by unsupervised training;The feature extraction part of rotation angle predictor is used as feature extractor;Object class predictor of RGB and depth image is constructed;Object class predictor is trained using image with label, and the result of RGB image or depth image semi-supervised classification is obtained;Then the prediction result of object class predictor is fused using the complementary information in RGB and depth image, the parameter of feature extractor is fine-tuned, so that the feature extraction part of rotation angle predictor adapts to the object classification task based on RGB image, depth image or RGB-D image.The application improves performance by depth mutual learning and fuses two kinds of modal specific object class predictor, after mutual learning, the object classification accuracy of RGB and depth image has been significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer pattern recognition and image classification technology, and in particular to a semi-supervised RGB-D object classification method based on angle prediction pre-training. Background Technology

[0002] With the development of 3D perception technology, 3D information has been widely used in object classification tasks. The 3D information of an object is typically converted into a depth map, where each pixel reflects the distance between the camera and points on the object's surface. This allows us to use both RGB images and depth maps simultaneously for object classification, often referred to as the RGB-D object classification task. Currently popular methods rely on machine learning, especially deep learning, which depends on a large number of labeled RGB and depth image pairs. However, usually only a small amount of annotated data is available. Therefore, it is crucial to train a classifier using both labeled and unlabeled paired RGB-D data, which can be addressed using the semi-supervised RGB-D Object Categorization (SSROC) algorithm. The key to this task is discovering complementary information between RGB and depth images and fully utilizing this information for model training, enabling RGB-D classification to outperform unimodal classification.

[0003] Currently, representative SSROC algorithms include: semi-supervised learning for RGB-D object recognition proposed in the paper [Yanhua Cheng, Xin Zhao, Kaiqi Huang, and Tieniu Tan. Semi-supervised learning for RGB-D object recognition[C]. In 2014 22nd International Conference on Pattern Recognition, pages 2377-2382, 2014.]; semi-supervised learning and feature evaluation for RGB-D object recognition[J]. Computer Vision and Image Understanding, 139: 149-160.]; and semi-supervised learning and feature evaluation proposed in the paper [Yanhua Cheng, Xin Zhao, Kaiqi Huang, and Tieniu Tan. Semi-supervised multi355 modal deep learning for RGB-D object recognition[C]. In Proceedings of the The semi-supervised multi355 modality deep learning approach proposed in [Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16, page 33453351. AAAIPress, 2016.] directly utilizes CNN-SPM-RNN for feature extraction and feature evaluation. The semi-supervised learning for RGB-D object recognition and the semi-supervised multi355 modality deep learning approach employ pre-training schemes to leverage unlabeled data. The semi-supervised learning for RGB-D object recognition uses an unsupervised convolutional-recurrent neural network for feature extraction, extracting a fixed tree structure from RGB-D images. Trees are originally used to describe scene images, but object images often lack complex structural information, so they are not well-suited for object images. The semi-supervised multi355 modality deep learning approach trains a reconstruction network using all RGB-D pairs and builds a classifier based on the reconstruction network.The output of the reconstruction network is assumed to be similar to its input data, thus the reconstruction network learns some hidden information from the training data that may be useful for classification. However, it is unclear what kind of information can be learned and how object classification will benefit from this information. In the decision-making phase, all work employs a co-training scheme. Semi-supervised multi355 modality deep learning is used simultaneously with classifiers for both categories and attributes to select high-confidence samples to generate pseudo-labels and expand the labeled training dataset. However, co-training requires two classifiers of specific modalities to select training samples to expand each other's training sets, and a classifier of one modality may not adapt well to the selected samples of the other modality. This typically reduces the effectiveness of co-training.

[0004] In recent years, Self-Supervised Learning (SSL) has been proposed to address semi-supervised learning tasks. SSL relies on camouflage tasks to learn information from unlabeled data samples in an unsupervised manner. The learned information has proven beneficial for various downstream tasks, such as object segmentation or object classification. The paper [Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E Hinton. A simple framework for contrastive learning of visual representations[C]. In Proceedings of 2008 International Conference on Machine Learning (ICML), pages 1597-1607. AAAIPress, 2020.] proposes contrastive learning as an effective camouflage task. It augments the dataset through color distortion, geometric distortion, cropping, and filtering of images, minimizing the distance between similar samples (augmented images from the same image) and maximizing the distance between dissimilar samples (augmented images from different images). However, due to color distortion in depth images, contrastive learning is difficult to apply to depth images. The unsupervised representation learning based on image rotation prediction proposed in the paper [Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations[C]. CoRR, abs / 1803.07728, 2018.] reveals that rich structural information can be learned by simply rotating the image to a pre-specified angle and training a classifier to predict the angle. This is because the model must know the shape, contour, and orientation of objects in the image to achieve accurate angle prediction. We believe that this structural information is crucial for object classification.Furthermore, the deep mutual learning proposed in the paper [Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning[C]. In Proceedings of 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4320-4328. IEEE, 2018.] demonstrates that the classification accuracy of multiple classifiers can be further improved by having each classifier simulate the predictions of other classifiers. This is because the prediction results consist of the probability that an object belongs to each category, which provides the classifiers with exceptional inter-class similarity that the one-hot vectors of the training labels cannot reflect. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a semi-supervised RGB-D object classification method based on angle prediction pre-training.

[0006] To achieve the above objectives, this invention provides a semi-supervised RGB-D object classification method based on angle prediction pre-training, comprising the following steps:

[0007] (1) Rotate labeled and unlabeled RGB-D images by 90 degrees, 180 degrees and 270 degrees respectively, and train the rotation angle predictor of RGB image and depth image respectively to obtain the network model trained by unsupervised training.

[0008] (2) Fix the feature extraction part of each rotation angle predictor in step (1) and use the feature extraction part as the feature extractor; construct object category predictors for RGB images and depth images, and then connect the object category predictors with the feature extractors to obtain a neural network model; then use labeled RGB images or depth images to classify and train the object category predictors to obtain the semi-supervised classification results of RGB images or depth images.

[0009] (3) The prediction results of two object category predictors are fused by using the complementary information contained in the RGB image and the depth image. When training each object category predictor and fusing the two object category predictors, the parameters of the feature extractor are fine-tuned so that the feature extraction part of each rotation angle predictor is adapted to object classification tasks based on RGB images, object classification tasks based on depth images or object classification tasks based on RGB-D images.

[0010] Further, step (1) includes the following sub-steps:

[0011] (1.1) Labeled and unlabeled RGB-D images are used as training images; the training images are rotated to 90 degrees, 180 degrees and 270 degrees in sequence to form an augmented image set, and the original images with a rotation angle of 0 are merged into this set to obtain a training set covering 4 rotation angles;

[0012] (1.2) The rotation angle predictor is trained using the training set obtained in step (1.1) through the first cross-entropy loss function, resulting in a network model trained in an unsupervised manner.

[0013] Furthermore, step (2) includes the following sub-steps:

[0014] (2.1) Fix the feature extraction part of each rotation angle predictor and use the feature extraction part as the feature extractor; activate the fully connected layer through batch normalization and ReLU activation function, and then construct the object category predictor through softmax function; connect the object category predictor with the feature extractor to obtain the neural network model; then, optimize the parameters of the fully connected layer using the first cross-entropy loss function using labeled RGB images or depth images.

[0015] (2.2) Constructing training sets for RGB and depth images: Assume the training set for RGB images is X = {X...} l ,X u},in The training set represents labeled RGB images. Let Y represent the training set of unlabeled RGB images; assuming the training set of depth images is represented as Y = {Y...} l ,Y u},in The training set representing labeled depth images, The training set representing unlabeled depth images;

[0016] (2.3) The training set X based on the labeled RGB images from steps (2.1) and (2.2) l The training set Y of labeled depth images l The object category predictor is trained by optimizing the fully connected layer parameters using a second cross-entropy loss through labeled RGB or depth images.

[0017] Furthermore, step (3) includes the following sub-steps:

[0018] (3.1) Using the object category predictor of the RGB image and depth image from step (2.1) and the training set X from (2.2) l Y lCalculate the class probabilities of RGB images and depth images respectively. Use the class probabilities of RGB images and depth images to calculate the kl-divergence that matches the probability estimates of the object class predictors of RGB images with those of the object class predictors of depth images, and the kl-divergence that matches the probability estimates of the object class predictors of depth images with those of the object class predictors of RGB images.

[0019] (3.2) The unsupervised loss of the kl-divergence from step (3.1) and the mutual learning between the RGB image and the depth image is used as the total loss function for training the neural network to train the object category predictor until convergence.

[0020] The beneficial effects of this invention are as follows: This invention utilizes unlabeled training images from each modality through self-supervised learning and achieves multimodal decision fusion through deep mutual learning. For each modality, labeled and unlabeled RGB or depth images are rotated to a fixed angle, and an angle predictor is trained to predict the angle. Then, the feature extraction part of the angle predictor is used to construct a category predictor, which is trained on labeled data. The object category predictors of the two modalities are fused by causing each modality to simulate the probability estimate generated by the other modality and making the prediction of the labeled image close to the true label. During the training and mutual learning of the object category predictor, the parameters of the feature extractor can be fine-tuned to avoid underfitting. This invention improves performance by fusing two modality-specific object category predictors through deep mutual learning. After mutual learning, the object classification accuracy of both RGB and depth images is significantly improved. Experimental results on the Washington RGB-D dataset show that this framework outperforms existing methods. Attached Figure Description

[0021] Figure 1 This is a framework diagram for semi-supervised training of RGB or depth images in this invention;

[0022] Figure 2 This is a schematic diagram illustrating the fusion training of RGB and depth images at the decision layer using kl-divergence in this invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0024] A common problem in RGB-D object classification is the lack of sufficient annotated training data. To address this, this invention proposes a semi-supervised RGB-D object classification method based on angle prediction pre-training, see [link to relevant documentation]. Figure 1 and Figure 2 This includes the following steps:

[0025] (1) Rotate labeled and unlabeled RGB-D images (i.e., RGB images and depth images) by 90 degrees, 180 degrees, and 270 degrees respectively, and train rotation angle predictors for RGB images and depth images respectively to obtain unsupervised trained network models, such as... Figure 1 The upper part. Specifically, it includes the following sub-steps:

[0026] (1.1) Labeled and unlabeled RGB-D images are used as training images; the training images are rotated sequentially to 90 degrees, 180 degrees, and 270 degrees to form augmented image sets, and the original images (with a rotation angle of 0) are merged into these sets to obtain a training set covering four rotation angles. Let x i For a single image in the training set, x i The angle label is a one-hot vector z. i =[z i (1), z i (2), z i (3), z i (4)];When vector z i When (j) = 1, it means that j represents the actual rotation angle, and when z i When (j) = 0, it means that j is not its actual rotation angle.

[0027] (1.2) The rotation angle predictor is trained using the training set obtained in step (1.1) with the first cross-entropy loss function. A convolutional neural network (CNN) consisting of some convolutional (Conv) blocks and fully connected (FC) layers is used for training, with batch normalization and ReLU activation function, as follows: Figure 1 As shown. Let p i =[p i (1), p i (2), p i (3), p i (4)] is the probability estimate generated by the softmax function, then the first cross-entropy loss function is loss angle (That is, the cross-entropy loss function for angle prediction) can be expressed as:

[0028]

[0029] In the formula, N l N represents the number of labeled RGB or depth images. u Indicates the number of unlabeled RGB or depth images.

[0030] Through steps (1.1) to (1.2), the network model trained in unsupervised manner is obtained.

[0031] (2) Fix the feature extraction part of each rotation angle predictor in step (1) and use this feature extraction part as the feature extractor; construct object category predictors for RGB images and depth images, and connect the object category predictors with each feature extractor to obtain a neural network model; train the object category predictors to classify using labeled RGB or depth images to obtain the semi-supervised classification results of RGB or depth images. Specifically, the following sub-steps are included:

[0032] (2.1) Once the training of the rotation angle predictor converges, fix the feature extraction part of each rotation angle predictor (the last convolutional layer and the previous network layers in the neural network, i.e.) Figure 1 The Conv part of the neural network is used as the feature extractor. The fully connected (FC) layers are activated using batch normalization and the ReLU activation function, and then a softmax function is used to construct an object category predictor. The object category predictor is connected to the feature extractor to obtain the neural network model. Then, labeled RGB or depth images are used to optimize the parameters of the fully connected (FC) layers of their respective neural networks using the first cross-entropy loss function.

[0033] Assumption For labeled RGB or depth images, regarding The features output by the last convolutional layer are The corresponding object category predictor can then be described as:

[0034]

[0035] in, express W utilizes the output of a fully connected layer and batch normalization. T This indicates a fully connected layer, and T indicates transpose. This represents a labeled RGB or depth image. This represents the feature output of a labeled RGB or depth image processed by a convolutional neural network, where BN stands for batch normalization. W represents the probability of the category, W is the weight parameter, and b is the offset parameter.

[0036] (2.2) Constructing training sets for RGB and depth images: Assume the training set for RGB images is X = {X...} l ,X u},in The training set represents labeled RGB images. This represents the training set of unlabeled RGB images. Assume the training set of depth images is represented as Y = {Y...} l ,Y u},in The training set representing labeled depth images, X represents the training set of unlabeled depth images. l and Y l The annotation is Where v i It is a c-dimensional one-hot vector, if or If it belongs to class j, then otherwise

[0037] (2.3) Based on the neural network model in step (2.1) and the training set X of labeled RGB images in step (2.2) l The training set Y of labeled depth images l The object category predictor is trained by optimizing the parameters of the fully connected (FC) layer using labeled RGB or depth images via a second cross-entropy loss function. The second cross-entropy loss function is Loss. class (That is, the cross-entropy loss function for object classification) can be expressed as:

[0038]

[0039] In the formula, C represents the number of object categories.

[0040] Based on formula (2), it is easy to construct more object category predictors for FC layers.

[0041] Through steps (2.1) to (2.3), the results of semi-supervised classification of RGB or depth images are obtained.

[0042] (3) Using deep mutual learning technology, the prediction results of two object category predictors are fused by utilizing the complementary information contained in RGB images and depth images. When training each object category predictor and fusing the object category predictors of RGB images and depth images, the parameters of the feature extractor are fine-tuned so that the feature extraction part is adapted to object classification tasks based on RGB images / depth images or RGB-D object classification tasks. Specifically, the following sub-steps are included:

[0043] (3.1) Using the object category predictor of the RGB and depth images from step (2.1) and the training set X from (2.2) l Y lCalculate the class probabilities for both the RGB and depth images. Then, use these probabilities to calculate the Kl-divergence between the probability estimates of the object class predictors for the RGB and depth images, respectively, to determine the matching Kl-divergence between the object class predictors for the depth and RGB images. Figure 2 As shown;

[0044] Deep mutual learning is employed to discover complementary information between two modality-specific object category predictors and fuse them to improve performance. The estimated vectors of the object category predictors are not one-hot vectors; rather, each element of the estimated vector represents the probability that the current sample belongs to a specific category. Therefore, the output of the object category predictors contains information representing the similarity between categories, which is missing in the true one-hot category labeling. The key to mutual learning is to allow the two modality-specific object category predictors to simulate the probabilities estimated by the other predictor, so as to discover the correct similarity between categories through the complementary efforts of the two predictors.

[0045] Specifically, the feature extraction parts of the modality-specific object category predictors are fixed, and their respective feature extractors are linked to multiple fully connected (FC) layers, activated by batch normalization and ReLU activation, and then passed through a softmax function. Mutual learning is achieved by minimizing the bidirectional KL divergence between the probabilities generated by the softmax function. To ensure that the results of mutual learning do not deviate from the true values, and to expect that the output of each modality-specific object category predictor is close to the labeling of these labeled data, the following operations are performed:

[0046] make This represents the features of an RGB image, where the labeled image is... Features representing depth images, where labeled images are A single-layer class probability estimate with batch normalization and ReLU activation can be described as follows:

[0047]

[0048]

[0049] in, Let represent the class probabilities of the RGB image and the depth image, respectively. The kl-divergence that matches the probability estimates of the object class predictors for the RGB image with those for the depth image is:

[0050]

[0051] Similarly, the kl-divergence that matches the probability estimates of the object class predictors for depth images with those for RGB images is:

[0052]

[0053] Therefore, the unsupervised loss method for mutual learning between RGB images and depth images is used. unsup It can be represented as:

[0054]

[0055] α1 and α2 are hyperparameters.

[0056] (3.2) The unsupervised loss of the kl-divergence from step (3.1) and the mutual learning between the RGB image and the depth image is used as the total loss function for training the neural network to train the object category predictor until convergence.

[0057] Supervision loss of mutual learning sup It can be represented as:

[0058]

[0059] in and It is an estimated class probability of labeled data.

[0060] Therefore, the total loss from the mutual learning of modality-specific category predictor variables is... mutual for:

[0061] Loss mutual =Loss sup +Loss unsup (9)

[0062] Multiple FC layers can be similarly constructed to learn from each other.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0064] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A semi-supervised RGB-D object classification method based on angle prediction pre-training, characterized in that, Includes the following steps: (1) Rotate labeled and unlabeled RGB-D images by 90 degrees, 180 degrees and 270 degrees respectively, and train the rotation angle predictor of RGB image and depth image respectively to obtain the network model trained by unsupervised training. (2) Fix the feature extraction part of each rotation angle predictor in step (1) and use the feature extraction part as the feature extractor; Construct object category predictors for RGB and depth images, then connect the object category predictors to a feature extractor to obtain a neural network model; then train the object category predictors using labeled RGB or depth images to obtain semi-supervised classification results for RGB or depth images; including the following sub-steps: (2.1) Fix the feature extraction part of each rotation angle predictor and use the feature extraction part as the feature extractor; activate the fully connected layer through batch normalization and ReLU activation function, and then construct the object category predictor through softmax function; connect the object category predictor with the feature extractor to obtain the neural network model; then, use labeled RGB images or depth images to optimize the parameters of the fully connected layer through the first cross-entropy loss function; (2.2) Constructing training sets for RGB and depth images: Assume the training set for RGB images is... ,in The training set represents labeled RGB images. Let represent the training set of unlabeled RGB images; assuming the training set of depth images is represented as . ,in The training set representing labeled depth images, The training set representing unlabeled depth images; (2.3) Training set based on labeled RGB images from steps (2.1) and (2.2) X l Training set of labeled depth images Y l The object category predictor is trained by optimizing the fully connected layer parameters using a second cross-entropy loss through labeled RGB or depth images. (3) Utilize the complementary information contained in the RGB image and the depth image to fuse the prediction results of the two object category predictors. When training each object category predictor and fusing the two object category predictors, fine-tune the parameters of the feature extractor so that the feature extraction part of each rotation angle predictor is adapted to object classification tasks based on RGB images, object classification tasks based on depth images, or object classification tasks based on RGB-D images; including the following sub-steps: (3.1) Using the object category predictor of the RGB image and depth image from step (2.1) and the training set from (2.2). X l , Y l Calculate the class probabilities of RGB images and depth images respectively. Use the class probabilities of RGB images and depth images to calculate the kl-divergence that matches the probability estimates of the object class predictors of RGB images with those of the object class predictors of depth images, and the kl-divergence that matches the probability estimates of the object class predictors of depth images with those of the object class predictors of RGB images. (3.2) The unsupervised loss of the kl-divergence from step (3.1) and the mutual learning between the RGB image and the depth image is used as the total loss function for training the neural network to train the object category predictor until convergence.

2. The semi-supervised RGB-D object classification method based on angle prediction pre-training according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1.1) Labeled and unlabeled RGB-D images are used as training images; the training images are rotated to 90 degrees, 180 degrees and 270 degrees in sequence to form an augmented image set, and the original images with a rotation angle of 0 are merged into this set to obtain a training set covering 4 rotation angles; (1.2) The rotation angle predictor is trained using the training set obtained in step (1.1) through the first cross-entropy loss function, resulting in a network model trained in an unsupervised manner.

Images