A self-supervised based rgb-d feature learning method

Abstract

A kind of self-supervised based RGB-D feature learning method, comprising the following steps: 1) processing training sample stage;2) network configuration stage;3) training stage;4) test stage.The self-supervised design framework designed in the application effectively combines Depth and RGB, and forces the self-supervised model to learn the shared information between modalities through contrastive learning and generative learning, thereby obtaining effective feature representation and better serving downstream tasks.Compared with the current method, using RGB pre-training model to initialize Depth branch model has modal difference, and using pixel-level constraint has the problem of uneven sample distribution.The application effectively fuses Depth and RGB information through self-supervised learning and contrastive learning, and avoids such problems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer software, and relates to self-supervised feature learning and scene classification, in particular to an RGB-D feature learning method based on self-supervision. BACKGROUND

[0002] In the field of computer vision, conventional supervised learning needs long-time annotation of a data set, which consumes a lot of resources and time, and self-supervised feature learning is to design a supervision signal by using the characteristics of the data itself, thereby avoiding the annotation of the data set. Self-supervised feature learning uses a large amount of unlabeled data to design a supervision signal by using the characteristics of the data itself, trains an effective self-supervised pre-training model, and uses the model to initialize a downstream task, so that the performance of the downstream task reflects the performance of the self-supervised feature learning.

[0003] In self-supervised feature learning, a reasonable proxy task is crucial. Early researchers obtain a supervision signal by processing an input image. Zhang et al. [2] An input grayscale image is used, and the original color information of the image is used as a supervision signal to train the model to understand the semantic information in the image and accurately color the image. Doersch et al. [3] The proxy task designed by Pathak et al. [4] fills in the missing part of the input image. In recent years, contrastive learning has attracted more and more attention. The DIM (Deep InfoMax) method [5] maximizes the connection between local features and global features of the same image to learn features, so that the model has the ability to distinguish between same-class samples and different-class samples. The SimCLR (Simple Contrastive Learning Representation) method [6] maximizes the consistency between different enhanced views of the same sample to learn visual representation. With the vigorous development of self-supervised learning methods, the performance of the methods in some fields has reached or even exceeded that of supervised learning methods.

[0004] Currently, the method combining RGB and Depth mainly uses a two-branch fusion strategy, but the Depth branch lacks a corresponding initialization model such as the RGB initialization model ImageNet, Places, etc., which often affects the training of the Depth branch by using the RGB pre-training model for initialization, thereby failing to effectively use the Depth information.

[0005] References

[0006] [1] Nowozin S, Cseke B, Tomioka R. f-gan: Training generative neural samplers using variational divergence minimization [C] / / Advances in neural information processing systems. 2016: 271-279.

[0007] [2] Zhang R, Isola P, Efros AA. Colorful image colorization [C] / / European conference on computer vision. 2016: 649-666.

[0008] [3] Pathak D, Kraehenbuh! P, Donahue J, et al. Context encoders: Feature learning by inpainting [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2536-2544.

[0009] [4] Doersch C, Gupta A, Efros AA. Unsupervised visual representation learning by context prediction [C] / / Proceedings of the IEEE international conference on computer vision. 2015: 1422-1430.

[0010] [5] Hjelm R D, Fedorov A, Lavoie-Marchildon S, et al. Learning deep representations by mutual information estimation and maximization [J]. arXiv preprint arXiv: 1808.06670, 2018.

[0011] [6] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations [C] / / International conference on machine learning. 2020: 1597-1607.

[0012] [7] DU D, WANG L, WANG H, et al. Translate-to-Recognize Networks for RGB-D Scene Recognition [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 11836-11845. SUMMARY

[0013] The problem to be solved by the present application is: how to effectively combine RGB data and Depth data, and then learn a reasonable framework based on a self-supervised method to learn effective feature representation. In order to effectively utilize the Depth information, the present application uses RGB / Depth through modal translation to interact with Depth / RGB images based on self-supervised learning, so as to improve the performance of the downstream scene classification task.

[0014] The technical scheme of the present application is: a self-supervised based RGB-D feature learning method, which combines RGB images and depth images to obtain RGB-D features, and constructs a supervised scene classification model using the RGB-D features for scene classification.

[0015] First, a self-supervised feature learning framework is trained, including a feature generator, a cross-modal feature extractor, an inter-structure discriminator and an intra-structure discriminator, the self-supervised feature learning framework takes an RGB image or a Depth image as an input modality, and takes a corresponding Depth image or RGB image as a target modality, first, the feature generator is used to perform down-sampling encoding and up-sampling decoding operations on the input modality to obtain a generated modality, then the cross-modal feature extractor is used to extract high-level semantic features from the generated modality and the corresponding target modality and a random target modality, respectively, the high-level semantic features of the generated modality are spliced with the high-level semantic features of the two target modalities as positive and negative samples to be input into the inter-structure discriminator for contrast learning, the shared high-level semantic information between the target modality and the generated modality is maximized, and the generated modality and the corresponding target modality are input into the intra-structure discriminator for generation learning based on the principle of the generative adversarial network, so that the distribution of the generated modality and the corresponding target modality data tends to be consistent, and the detailed information of the target modality is learned;

[0016] According to the feature generator in the self-supervised feature learning framework obtained by training, two layers of randomly initialized fully connected layers are connected to obtain a supervised scene classification model network structure, and then the supervised scene classification model network structure is trained and fine-tuned by training samples to obtain a final supervised scene classification model.

[0017] Further, the present application comprises the following steps:

[0018] 1) Processing training samples, the training samples include RGB images and corresponding Depth images, and the single-channel Depth images are converted into three-channel color images in HHA format, H, H and A represent horizontal difference, height above ground and angle of surface normal vector, respectively;

[0019] 2) Network model configuration stage, first, a self-supervised feature learning framework is constructed, taking an RGB image as an input modality and a corresponding Depth image as a target modality, or taking a Depth image as an input modality and a corresponding RGB image as a target modality, a generated modality is obtained from the input modality, interacts with the target modality, and a supervised scene classification model is obtained from the trained self-supervised feature learning framework; the self-supervised feature learning framework comprises the following configurations:

[0020] 2.1) Feature generator, using ResNet18 as the backbone network, performing down-sampling encoding on the input features, and then performing up-sampling decoding to obtain the generated modality features, wherein, in the first convolution operation of the backbone network layer1, a convolution operation with a step of 2 is used for down-sampling;

[0021] 2.2) Cross-modal feature extractor, the generated modal and the corresponding target modal, and a random target modal, are taken as inputs of the cross-modal feature extractor respectively, and high-level semantic features, i.e. high-dimensional features, are extracted respectively;

[0022] 2.3) Inter-structure discriminator, the high-dimensional features of the two target modalities are respectively spliced with the high-dimensional features of the generated modal to form two positive and negative samples, and then dimension reduction is performed on the positive and negative samples by using a 1*1 convolution layer, and then contrastive learning is performed;

[0023] 2.4) Intra-structure discriminator, the intra-structure discriminator is a generative adversarial network, and the intra-structure discriminator takes the high-dimensional features of the generated modal and the corresponding target modal high-dimensional features as inputs, so that the generated modal learns the high-dimensional feature information of the corresponding target modal;

[0024] 2.5) Supervised scene classification model: the self-supervised feature learning framework configured by steps 2.1) to 2.4) is trained by the training samples of step 1), and two layers of randomly initialized fully connected layers are added to the obtained backbone network ResNet18 to obtain a supervised scene classification model;

[0025] 3) Training phase, the supervised scene classification model is initialized based on the parameters of the trained self-supervised feature learning framework, and the supervised scene classification model is further fine-tuned by using the training samples;

[0026] 4) Test phase, for the image to be classified, the trained supervised scene classification model is used for classification.

[0027] The object of the application is to construct a reasonable and effective self-supervised feature learning agent task using RGB images and Depth (depth) images, so that the backbone network of the agent task has the ability to distinguish between homogeneous samples and heterogeneous samples, thereby better serving downstream tasks. Unlike the prior art, which only focuses on a single modal data design agent task, resulting in suboptimal results. The application learns effective features by combining RGB images and Depth images. Du et al [7]Based on the supervised learning method, the RGB image and the depth image are fused, a ResNet18 model based on ImageNet training is used to generate the depth image and the real depth image using L2 loss for pixel-level constraint for guiding modal translation.

[0028] Compared with the prior art, the present application has the following advantages.

[0029] The present application proposes a self-supervised RGB-D feature learning method, which effectively utilizes RGB and Depth based on a self-supervised feature learning framework, and maximizes the shared structural information between the generated modal and the target modal through the contrast learning module and the generation learning module, and makes the distribution of the generated modal consistent with that of the target modal.

[0030] The present application effectively combines RGB images and depth images, learns the shared structural information between them, and obtains effective features to better serve downstream tasks.

[0031] The self-supervised pre-training model designed by the present application can obtain good performance when used for scene classification task initialization. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a self-supervised feature learning framework diagram of the present application.

[0033] Figure 2is the first layer convolutional schematic diagram of layer1 in Resnet18 in the network structure of the application.

[0034] Figure 3 is the structure schematic diagram of "convolution normalization activation" in the upsampling decoder used in the application.

[0035] Figure 4 is the structure schematic diagram of the cross-modal feature extractor.

[0036] Figure 5 is the structure schematic diagram of the inter-structure discriminator.

[0037] Figure 6 is the structure schematic diagram of the intra-structure discriminator.

[0038] Figure 7 is the schematic diagram of the supervised scene classification model.

[0039] Figure 8 is the sample diagram of the generated modal in the self-supervised pre-training process of the application. DETAILED DESCRIPTION

[0040] The application proposes an RGB-D feature learning method based on self-supervision. After pre-training on DIML, SUN RGB-D and NYUD2 data sets, fine-tuning on the training sets of SUN RGB-D and NYUD2 data sets, and reaching the most advanced self-supervised learning performance on the test sets of the two data sets, the specific Python3.6 programming language and Pytorch1.1 deep learning framework are used.

[0041] Figure 1 is the self-supervised feature learning framework diagram used in the application, the application uses a residual network ResNet18 as a backbone network, combines RGB images and depth images, reduces the difference between input modalities and target modalities through modal translation. The application learns shared high-level semantic information between data modalities and detailed information within data modalities based on contrast learning and generative learning for self-supervised learning. The whole method includes processing training sample stage, network configuration stage, training stage and testing stage. The specific implementation steps are as follows:

[0042] 1) Processing training sample stage, the training sample in the pre-training stage of the application includes an RGB image and a corresponding depth image. In order to better correspond to the RGB image, the single-channel depth image is first converted into a three-channel color image in HHA format, and H, H and A represent the horizontal difference, the height above the ground and the angle of the surface normal vector, respectively. The depth image and the RGB image and their corresponding semantic labels are used to fine-tune the downstream task. The application uses 7226 RGB-D data frames extracted from the video dataset DIML, and the training set of SUN RGB-D and the training set of NYUD2 to pre-train, and then trains on the training set of SUN RGB-D and NYUD2 in the fine-tuning supervised model stage, and tests on the corresponding test set after training.

[0043] 2) Model configuration stage, that is, network configuration stage, a self-supervised feature learning framework is constructed, including four parts, a feature generator, a cross-modal feature extractor, an inter-structure discriminator and an intra-structure discriminator, wherein G β represents the feature generator, and β represents the parameters of G β , C φ represents the cross-modal feature extractor, and φ represents the parameters of C φ , D ω represents the inter-structure discriminator, and ω represents the corresponding parameters of D ω , D α represents the intra-structure discriminator, and α is the corresponding parameter of D α . The application uses a residual network ResNet18-based autoencoder as the backbone network to construct a self-supervised feature learning framework, taking the RGB image and the depth image as the input modality and the target modality, obtaining the generated modality from the input modality, interacting with the target modality, completing the pre-training, and serving the downstream task. In the application, the RGB image can be used as the input modality, and the corresponding depth image is the target modality, or the depth image can be used as the input modality, and the corresponding RGB image is the target modality, Figure 1 shows the case when the RGB image is used as the input modality, and the depth image is used as the input modality, which is consistent with its structure, except that the target modality is different. As Figure 1As shown, the self-supervised learning system of the present application first uses the feature generator to perform up-sampling encoding and down-sampling decoding operations on the input modalities, obtains the generated modalities, and then uses the generated modalities and corresponding target modalities and a random target modality to extract high-level semantic features using the cross-modal feature extractor, respectively. Then, the generated modalities and the two target modality features are spliced as positive and negative samples, respectively, and input into the inter-structure discriminator. Based on the contrast learning module, the shared high-level semantic information between the target modalities and the generated modalities is maximized. At the same time, the generated modalities and the corresponding target modalities are input into the intra-structure discriminator, and based on the generated learning, the distribution of the generated modalities and the corresponding modal data tends to be consistent, so as to learn the detailed information of the target modalities. The specific structure is as follows:

[0044] 2.1) Feature generator, the input modality format is x i = B * 3 * 224 * 224, where B represents the batch size Batchsize. First, down-sampling encoding operation is performed using the backbone network ResNet18. ResNet18 is composed of layer0, layer1, layer2, layer3, layer4 and a fully connected layer FC, and is pre-trained. Layer0 only contains one convolutional layer, and the remaining layers all contain four convolutional layers. The present application uses a structure as Figure 2 instead of the convolution operation of the first layer of layer1 of Resnet18. The convolution operation with a step size of 2 is used instead of the Maxpooling operation directly used in the original ResNet18 structure. The present application uses the features processed by Maxpooling as a residual module to participate in the training of the feature generator. Based on this structure, the information loss caused by Maxpooling operation is effectively reduced, more useful information is retained, and the subsequent feature decoding operation is beneficial. In layer0 of ResNet18, the input B * 3 * 224 * 224 is used for the first time dimension reduction using a convolution kernel size of 7 * 7 and a convolution operation with a step size of 2 to obtain an output feature of B * 64 * 112 * 112. Then, the output feature of layer0 is sequentially passed through layer1, layer2, layer3 and layer4 to obtain feature maps of B * 64 * 56 * 56, B * 128 * 28 * 28, B * 256 * 14 * 14 and B * 512 * 7 * 7. Layer1, layer2, layer3 and layer4 all use a convolution kernel with a step size of 2 to achieve the purpose of reducing the size. Then, a structure as Figure 3The "convolution normalization activation" (ConvNormRelu) module decodes the features with an output size of B*512*7*7 of layer4. The decoder uses a "convolution normalization activation" (ConvNormRelu) as the basic unit of the decoder. Compared with the size of the input image, the features need to be restored from 1 / 32 size to 1 / 4 size, a total of three times, in turn 1 / 32→1 / 16, 1 / 16→1 / 8, 1 / 8→1 / 4. The specific steps are as follows: (a) first bilinear interpolation of the current feature; (b) perform "convolution normalization activation" on the feature corresponding to the size of the encoder, (c), splice the features of (a) and (b); (d) perform "convolution normalization activation" on the feature of (c). After three restoration operations, the final decoded feature with a size of B*3*56*56 is obtained, and a bilinear interpolation operation is performed again to obtain the generated modality y consistent with the size of the input modality i .

[0045] 2.2) Cross-modal feature extractor, the cross-modal feature extractor is composed of two layers of convolution, as shown in Figure 4 , using convolution operation with convolution kernel size of 3*3 and step of 2 to compress the size of the input feature and extract high-level semantic features. The present application uses the idea of contrast learning, so it is necessary to use the generated modality y i and the corresponding target modality x i , and a random target modality x j , the three groups of data are respectively input into the cross-modal feature extractor, first input into the first layer of convolution to obtain B*64*112*112, and then input into the second layer of convolution to obtain feature maps with a size of B*128*56*56 The generated modality and the target modality share the same feature extractor, which helps the model to learn the shared high-level semantic information of the generated modality and the target modality.

[0046] 2.3) Inter-structure discriminator, i.e. contrast learning module, aims to explore the shared high-level semantic information between the generated modality and the target modality, and maximize the shared high-level semantic information between the two through the contrast learning module, so that the backbone network has the ability to distinguish between related samples and unrelated samples, and the encoding of related samples is closer, and the encoding distance of unrelated samples is as far as possible. Different from the method of directly using Euclidean distance to measure the similarity of pictures, the purpose of the inter-structure discriminator in the present application is to interact between the generated modality and the target modality, so as to learn the shared high-level semantic information between the paired modal data. Similar to variational difference minimization, shared high-level semantic information maximization is also based on a given real distribution P, to estimate a generation model Q:

[0047] F(θ,ω)=E y～P[T ω (y)]-E y～Qθ [f*(T ω (y))],

[0048] where E denotes expectation, y denotes sample, T denotes extracting sample feature, T ω (y) denotes high-level semantic information extracted for corresponding sample y, f denotes f-GAN[1], a generative adversarial network GAN constructed by f divergence. P is a real distribution. Q is a generative model, and θ is the parameter corresponding to Q. Based on the above formula, the present application uses Jenson-Shannon mutual information to estimate the difference between positive and negative examples. In order to optimize the f-GAN objective function therein, the present application uses a small batch of generated features and the expectation of target modal features to approximate the overall expectation. For example, given a set of generated modalities y = {y1, y2, y3,...} and corresponding target modalities x = {x1, x2, x3,...}, the two sets of features are calculated using cross-modal feature extractors, respectively denoted as and The model is optimized as follows:

[0049]

[0050] where E denotes expectation, P denotes the distribution between corresponding samples in the generated modalities and the target modalities, and P denotes the distribution between non-corresponding samples in the generated modalities and the target modalities. denotes a positive example pair, denotes a negative example pair, i≠j, denotes a generated feature sample, denotes a corresponding sample in the target modalities, denotes a random sample in the target modalities. The inter-structure discriminator D ω is shown as follows: Figure 5 where Depth1' denotes the generated modalities, Depth1 denotes the Depth data in the target modalities corresponding to the input modalities, and Depth2 denotes randomly selected Depth data in the target modalities. Here, only the process when the input is RGB is shown, and the process when the input is Depth is similar. First, the feature map with an input size of B*256*56*56 is passed through three convolutional layers D ω with a kernel size of 1*1, and two groups of feature maps with sizes of B*512*56*56, B*512*56*56, and B*1*56*56 are obtained in turn, respectively. The final output of the positive and negative feature samples with two channels of 1 is then passed through a softplus operation, and the difference is obtained to get the final contrastive learning loss.

[0051] 2.4) Intra-structure discriminator, i.e. a generated learning module, aims to make the distribution of the generated modal and the target modal consistent. The internal structure between the generated modal and the target modal is similar, and the pixel-level information of the generated modal and the target modal is also consistent. The intra-structure discriminator learns the internal structure relationship between the generated modal and the corresponding target modal. The texture features often ignored in the modal translation process are important for complementary clue transfer. The intra-structure discriminator aims to help the generation network to generate more real data to match the distribution of the target modal. Compared with directly using a similar Euclidean distance constraint for pixel-level accurate matching, this way has greater gain for modal transfer. The invention uses a traditional generation loss for optimization, and the formula is as follows:

[0052]

[0053] L intra (α, β, φ | x, y) is the loss function of the intra-structure discriminator, x and y respectively represent the true sample and the false sample, that is, the target modal and the input modal, G β represents the feature generator, and the input modal y i is input into the feature generator to obtain the generated modal sample G β (y i ), C φ represents the cross-modal feature extractor, C φ (x i ) is the feature of the target modal sample x i obtained by the cross-modal feature extractor, D α represents the intra-structure discriminator, E is the expectation function, P x represents the target modal distribution, P y represents the input modal distribution. The intra-structure discriminator first inputs the generated modal feature and the target modal feature into the network structure as shown in Figure 6 , respectively. First, the generated Depth1' feature map with a size of B*128*56*56 is taken as negative sample data, and the corresponding target modal Depth1 is taken as positive sample data, which are respectively input into a network composed of three convolutional layers for feature extraction and dimension compression. The convolution kernel size of the three convolutional layers is 3*3, 3*3 and 1*1, respectively. After the feature input, three groups of features with sizes of B*256*56*56, B*512*56*56 and B*1*56*56 are obtained in turn. The positive sample feature with a channel number of 1 is calculated with the label 1 via the BCELoss to obtain the loss, and the negative sample feature with a channel number of 1 is calculated with the label 0 via the BCELoss to obtain the loss. The two losses are added to obtain the final intra-structure discriminator loss.

[0054] During training, the feature generator, cross-modal feature extractor and two discriminators are trained alternately. First, in order to detect false generated modalities, the intra-structure discriminator updates the parameters according to the formula in 2.4), and then the parameters of the intra-structure discriminator are fixed, and the parameters of the feature generator, the cross-modal feature extractor and the inter-structure discriminator composed of two formulas in 2.3) are updated, and the joint update of the parameters is as follows:

[0055] L total =λ1L inter +λ2L intra ,

[0056] Wherein, λ1 and λ2 are two hyperparameters during training, because the information between structures is more important, and it is difficult to optimize the loss L intra of the intra-structure discriminator when the gradient is large, therefore, in the specific experiment, the hyperparameters λ1 and λ2 are preferably set to 10 and 0.1 respectively.

[0057] 2.5) Construct a supervised scene classification model, the supervised scene classification model of the present application is as shown in Figure 7 The present application is a two-stage model, first, the self-supervised feature learning framework described in 2.1)-2.4) is constructed, and then the backbone network of the self-supervised feature learning framework is used to increase two layers of randomly initialized fully connected layers as a supervised scene classification model, as Figure 7 . Figure 7 The backbone network and parameters of Figure 1 , i.e. the down-sampling part in Figure 1 and Resnet18 in Figure 7 are the same module. The supervised scene classification model first extracts features based on the backbone network Resnet18, and then converts the features into a feature vector with a dimension of 1 and a length of the number of categories by using the FC layer, and then calculates the classification loss based on CrossEntropyLoss between the feature vector and the real label, wherein the real label is the category information artificially labeled in the public data set, and the prediction label is the maximum one-dimensional index (each dimension of the N-dimensional vector corresponds to a category) calculated by the softmax of the N*1 (N is the specific number of categories) dimensional feature vector finally output by the fully connected layer as the prediction label.

[0058] The self-supervised feature learning framework of the present application uses the inter-structure discriminator for contrastive learning and the intra-structure discriminator for generation learning to constrain the whole self-supervised feature learning framework, and then uses the backbone network parameters trained by the self-supervised feature learning framework to initialize the feature extraction module in the supervised scene classification model of Figure 7 , and further trains the supervised scene classification model by using the artificially labeled pictures.

[0059] The network configuration phase is specifically illustrated below with an example. The remaining structure of Resnet18 except the FC layer is used as the backbone network, which is randomly initialized, and then the input modal is encoded to obtain feature maps with sizes of B*64*112*112, B*64*56*56, B*128*28*28, B*256*14*14 and B*512*7*7 through layer0, layer1, layer2, layer3 and layer4 respectively, and then the encoder features are up-sampled and decoded. In the decoding process, the features are combined with the encoder features step by step by using the feature splicing method similar to Unet. First, the bilinear interpolation operation is performed on the encoder features of the last layer to restore the size of the previous layer to obtain the feature up3 with a size of B*512*14*14. Then, the layer3 encoder features are subjected to the "convolution normalization activation" (k=3, s=1) operation and spliced with the feature up3 in the channel dimension to obtain a feature with a size of B*768*14*14, and then subjected to twice "convolution normalization activation" (first k=1, s=1; second k=3, s=1) operation to obtain a feature map with a size of B*256*14*14. The bilinear interpolation is performed to obtain a feature map with a size of B*256*28*28 up2. The layer2 corresponding feature is spliced with up2 after the "convolution normalization activation" operation, and then subjected to twice "convolution normalization activation" operation to obtain a feature map with a size of B*256*28*28. The bilinear interpolation is performed to obtain a feature map with a size of B*256*56*56, which is spliced with the layer1 feature subjected to the "convolution normalization activation" operation, and then subjected to twice "convolution normalization activation" operation. Then, a convolution layer with an output channel number of 3 and a convolution kernel size of 1*1 is used to process the feature, and the bilinear interpolation is performed to obtain the generated modal with the same size as the input modal. The generated modal, the corresponding target modal and the random target modal are input into the cross-modal feature extractor respectively, and three groups of data with a size of B*128*56*56 are obtained after two convolution layers. The generated modal and the two target modals are spliced respectively, and two groups of feature maps with sizes of B*512*56*56, B*512*56*56 and B*1*56*56 are obtained through three convolution layers with a convolution kernel size of 1*1. The two positive and negative feature samples with a channel number of 1 are output finally, and the difference is obtained after the softplus operation to obtain the final contrast learning loss. Meanwhile, the generated modal feature map with a size of B*128*56*56 is taken as the negative sample data, and the corresponding target modal data is taken as the positive sample data, which are input into the network composed of three convolution layers for feature extraction and dimension compression.The convolution kernel size of the three-layer convolutional network is 3*3, 3*3 and 1*1 respectively, and three groups of features with sizes of B*256*56*56, B*512*56*56 and B*1*56*56 are obtained in sequence after feature input. The positive sample feature with a channel number of 1 and the label 1 are calculated by the BCELoss to obtain the loss, the negative sample feature with a channel number of 1 and the label 0 are calculated by the BCELoss to obtain the loss, and the two are added to obtain the final structure discriminator loss. When fine-tuning the supervised scene classification model, first, the feature is extracted based on the backbone network Resnet18, then the feature is converted into a feature vector with a dimension of 1 and a length of the number of categories by using the FC layer, and then the classification loss is calculated based on the CrossEntropyLoss between the feature vector and the real label.

[0060] 3) Offline training phase, since the present application learns effective features by maximizing the shared high-level semantic information between paired modal data, using more unlabeled RGB-D data to participate in training can improve the robustness of the model. In this way, the self-supervised model can migrate the pre-trained weights before training, selectively load a part of the weights during actual training, and keep the random initialization of the two discriminators. Loading the pre-trained weights can make the cross-modal feature extractor in the self-supervised learning framework generate high-quality generated images, and randomly initializing the two discriminators can force the self-supervised model to explore cross-modal information during training. Therefore, 7226 frames of RGB-D data extracted from the DIML dataset were used to participate in self-supervised pre-training during the experiment. DIML is a high-definition RGB-D video dataset, and in order to ensure uniformity, the present application samples at a frame rate of 20FPS during the experiment.

[0061] The experiment is based on the current popular machine learning framework Pytorch and Python language implementation, and uses 8 NVIDIA 2080Ti GPU cards. In the experiment, the input image is randomly cropped from 256*256 to 224*224, and randomly horizontally flipped with a probability of 0.5. The number of training batch pictures is set to 256, and all modules are trained using Adam except the structure discriminator which uses SGD. The learning rate is set to 0.0004, the number of iterations epochs is set to 500 on the SUN RGB-D and DIML datasets, and the number of iterations epochs is set to 100 on the NYUD2 dataset. The performance of the depth data encoded as Jet is equivalent to that of the HHA. The contrast learning loss is obtained by directly subtracting the features of the positive and negative samples for the inter-structure discriminator in the self-supervised pre-training model. The intra-structure discriminator uses BOELoss to calculate the loss based on positive and negative samples, and then adds the average value as the generation learning loss. The CrossEntropyLoss is used to calculate the loss of the supervised scene classification model, and the back propagation algorithm is used to update the network parameters.

[0062] The generated features of the SUN RGB-D dataset pre-training are shown in Figure 8 The left three columns are the generated results when RGB is used as the input modality, and the right three columns are the generated feature results when the Depth image is used as the input modality. From the figure, it can be observed that the generated image retains the features of the input modality and the target modality, i.e., it contains the texture and other features of the RGB image, and also contains the depth information of the Depth image. Therefore, it can be seen that the generated image has shared structural information between the two.

[0063] 4) Test phase, scale the picture to 256*256 and then use center cropping to obtain a 224*224 image as the input image. Input the trained supervised scene classification model, compare the predicted label with the real label, calculate the mean accuracy (Mean Accuracy), and do not calculate the loss and update the parameters. In terms of test accuracy, on the SUN RGB-D dataset, when RGB is used as the input modality, the test performance reaches an average accuracy of 44.4, and when Depth is used as the input modality, the test performance reaches an average accuracy of 42.6. On the NYUD2 dataset, when RGB is used as the input modality, the test performance reaches 53.3, and when Depth is used as the input modality, the test performance reaches 46.5. Both reach the state-of-the-art unsupervised performance, and only have a slight gap with supervised learning.

Claims

1. A self-supervised based RGB-D feature learning method, characterized in that RGB-D features are obtained by combining an RGB image and a depth image, and a supervised scene classification model is constructed using the RGB-D features for scene classification; First, a self-supervised feature learning framework is trained, The self-supervised feature learning framework includes a feature generator, a cross-modal feature extractor, an inter-structure discriminator and an intra-structure discriminator. The self-supervised feature learning framework takes an RGB image or a depth image as an input modality and takes a corresponding depth image or RGB image as a target modality. First, the feature generator is used to perform down-sampling encoding and up-sampling decoding on the input modality to obtain a generated modality. Then, the cross-modal feature extractor is used to extract high-level semantic features from the generated modality and the corresponding target modality and a random target modality. The high-level semantic features of the generated modality are concatenated with the high-level semantic features of the two target modalities, respectively, as positive and negative samples, which are input into the inter-structure discriminator for contrastive learning to maximize the shared high-level semantic information between the target modality and the generated modality. Meanwhile, the generated modality and the corresponding target modality are input into the intra-structure discriminator for generation learning based on the principle of generative adversarial networks to make the distribution of the generated modality and the corresponding target modality data consistent, so as to learn the detailed information of the target modality. According to the feature generator in the trained self-supervised feature learning framework, two layers of randomly initialized fully connected layers are connected to obtain a supervised scene classification model network structure. Then, the supervised scene classification model network structure is trained and fine-tuned by training samples to obtain a final supervised scene classification model. The method comprises the following steps: 1) processing training samples, the training samples including an RGB image and a corresponding depth image, and converting a single-channel depth image into a three-channel color image in HHA format, wherein H, H and A represent horizontal difference, height above ground and angle of surface normal vector, respectively; 2) network model configuration, first, a self-supervised feature learning framework is constructed, taking an RGB image as an input modality and a corresponding depth image as a target modality, or taking a depth image as an input modality and a corresponding RGB image as a target modality, obtaining a generated modality from the input modality, and interacting with the target modality to obtain a supervised scene classification model from the trained self-supervised feature learning framework; the self-supervised feature learning framework comprises the following configurations: 2.1) feature generator, using ResNet18 as a backbone network to perform down-sampling encoding and then up-sampling decoding on the input features to obtain a generated modality feature, wherein a convolution operation with a step of 2 is used for down-sampling at the first convolution operation of the backbone network layer1; 2.2) cross-modal feature extractor, the generated modality and the corresponding target modality, and a random target modality are input into the cross-modal feature extractor respectively to extract high-level semantic features, i.e., high-dimensional features; 2.3) inter-structure discriminator, the high-dimensional features of the two target modalities are concatenated with the high-dimensional features of the generated modality to form two positive and negative samples, and then a 1*1 convolution layer is used to reduce the dimension of the positive and negative samples for contrastive learning. 2.4) Intra-structure discriminator, the intra-structure discriminator is a generative adversarial network, the intra-structure discriminator takes the generated modality high-dimensional feature and the corresponding target modality high-dimensional feature as input, so that the generated modality learns the high-dimensional feature information of the corresponding target modality; 2.5) Supervised scene classification model: training the self-supervised feature learning framework configured by steps 1) to 2.4) by the training samples, adding two layers of randomly initialized fully connected layers to the obtained backbone network ResNet18 to obtain a supervised scene classification model; 3) Training phase, initializing the supervised scene classification model based on the parameters of the trained self-supervised feature learning framework, and further fine-tuning the supervised scene classification model using training samples; 4) Test phase, classifying the image to be classified by the trained supervised scene classification model.

2. The self-supervised based RGB-D feature learning method according to claim 1, characterized in that The network model is configured as follows: 2.1) a feature generator, with the input modality format being , first using the backbone network ResNet18 to perform down-sampling encoding operation, ResNet18 is composed of layer0, layer1, layer2, layer3, layer4 and a fully connected layer FC, wherein layer0 only contains one convolutional layer, and the remaining layers each contain 4 convolutional layers; after down-sampling, the output of layer4 is decoded using an up-sampling decoder to obtain a generated modality consistent with the size of the input modality ; wherein the input modality processed by Maxpooling is used as a residual module to participate in the training of the feature generator; 2.2) Cross-modal feature extractor, which consists of two layers of convolution, which compresses the size of the feature and extracts high-level semantic features. Based on the idea of contrastive learning, the generated modality is used and the corresponding target modality , and a random target modality The three groups of data are respectively input into the cross-modal feature extractor to obtain feature maps 、 、 ; 2.3) Inter-structure discriminator, learning the shared high-level semantic information between the pair of modal data by the interaction between the generated modality and the target modality, the inter-structure discriminator aims to maximize the shared high-level semantic information between the target modality and the generated modality, and the shared high-level semantic information is estimated based on a given real distribution P, a generation model Q: （1） is the objective function for estimating the generative model Q from the real distribution P, where E denotes expectation, y denotes a sample, denotes the high-level semantic information extracted from the corresponding sample y, f denotes f-GAN, i.e., a generative adversarial network GAN constructed using f-divergence, is the parameter corresponding to the generative model Q, and is obtained using denotes the inter-structure discriminator, is the corresponding parameter; Based on formula (1), the difference between positive examples and negative examples is maximized by using Jenson-Shannon mutual information estimation; 2.4) Intra-structure discriminator, the intra-structure discriminator learns the internal structural relationship between the generated modality and the corresponding target modality, and uses a generation loss for optimization, the formula is as follows: （2） is the loss function of intra-structure discriminator, x and y are the real and fake samples respectively, is the feature generator, and the input modality is is the generated modality sample after feature extraction , is the parameter of , is the cross-modal feature extractor, is the target modality sample is the feature obtained by the cross-modal feature extractor, is the parameter of is the intra-structure discriminator, is the is the corresponding parameter, and E is the expectation function, is the target modality distribution, is the input modality distribution; the intra-structure discriminator takes the generated modality feature and the target modality feature as input, first takes the generated modality Depth1' feature map as negative sample data, and the corresponding target modality Depth1 as positive sample data, and inputs them into the network composed of three layers of convolution, respectively, to perform feature extraction and dimension compression, and then calculates the loss of the positive sample feature with a channel number of 1 and the label 1 via the BCELoss, and the loss of the negative sample feature with a channel number of 1 and the label 0 via the BCELoss, and adds the two losses to obtain the final intra-structure discriminator loss. 2.5) Constructing a supervised scene classification model, the supervised scene classification model includes a feature extraction module and two fully connected layers, first initializing the feature extraction module based on the backbone network Resnet18 parameters of the self-supervised feature learning framework, which is used to extract features, then using the FC layer to convert the extracted features into a feature vector with a dimension of 1 and a length of the number of categories, then based on the training samples, using the feature vector and the real label based on CrossEntropyLoss to calculate the classification loss, and training the supervised scene classification model.

3. The self-supervised based RGB-D feature learning method according to claim 2, characterized in that 2.3) In order to optimize the f-GAN objective function in equation (1), the expected population expectation is approximated using a small batch of generated features and target modality features, given a set of generated modality and the corresponding target modality These two sets of features are computed using a cross-modal feature extractor, denoted as and respectively. The model optimization is shown as follows: （3） wherein E represents the distribution between the corresponding samples in the generation modal and the target modal, represents the distribution between the non-corresponding samples in the generation modal and the target modal, represents the positive example pair, represents the negative example pair, , represents the generation feature sample, represents the corresponding sample in the target modal, represents the random sample in the target modal, the input feature map of the inter-structure discriminator is passed through three convolutional layers , the positive and negative feature samples with two channels of 1 are passed through the softplus operation, and the difference is obtained to obtain the final contrast learning loss.

4. The self-supervised based RGB-D feature learning method according to claim 2, characterized in that during training, the feature generator, the cross-modal feature extractor, the inter-structure discriminator and the intra-structure discriminator are alternately trained, first, in order to detect false generated modalities, the intra-structure discriminator updates the parameters according to formula (1), then fixes the parameters of the intra-structure discriminator, and then updates the parameters of the feature generator, the cross-modal feature extractor and the inter-structure discriminator, the joint update of the parameters is as follows: wherein are two hyperparameters at training time.

5. The self-supervised based RGB-D feature learning method according to claim 4, characterized in that Setting hyperparameters and are 10 and 0.1, respectively.

Images