Indoor scene recognition method and system based on combined semantic region relationship model

Abstract

The application discloses an indoor scene recognition method and system based on a combined semantic region relationship model, and the method comprises the following steps: acquiring an indoor scene image to be recognized; inputting the acquired image into the combined semantic region relationship model, extracting semantic global node features of the input image through a semantic region relationship module, extracting global node features of the input image through an RGB feature extraction module, aggregating the semantic global node features and the global node features through a global feature aggregation module to obtain global aggregated node features, inputting the extracted global aggregated node features into a classifier, and outputting a scene recognition result. The application constructs a semantic region relationship model, fully mines the relationship between semantic regions of objects in an indoor scene, and effectively processes the semantic segmentation error problem in the model; the combined semantic region relationship model is constructed, the complementary information between features is more effectively mined, and the indoor scene recognition with better accuracy and generalization performance is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene recognition, and particularly relates to an indoor scene recognition method and system based on a combined semantic region relationship model. BACKGROUND

[0002] Indoor scene recognition is one of the important research topics in computer vision, and is widely used in application fields such as intelligent cameras and intelligent robots. Since it can provide a basic description of image content, scene recognition is also considered as prior knowledge for other computer vision tasks, such as image retrieval and object detection.

[0003] Early indoor scene recognition methods use local visual descriptors, which are integrated into image representations through the BOVW framework. Quattoni et al. proposed a prototype-based indoor scene model that can combine local and global discriminative information. However, the features used in this method are all hand-crafted, and are limited for distinguishing scenes that are ambiguous or highly similar.

[0004] In recent years, deep neural networks (DNNs) have made great progress in computer vision tasks, and can explore high-level representations of images. In recent years, scene recognition methods based on DNNs have achieved excellent performance. For example, Dual CNN-DL proposed a new dictionary learning layer to replace the traditional FCL and ReLu, and enhanced the sparse representation and discriminative ability of features by determining the optimal dictionary; Lin et al. proposed a hierarchical coding algorithm to convert convolutional features into the final image representation for scene recognition. The above methods use convolutional neural networks to extract scene representations, which greatly improves the recognition results, but there is still a big gap from the results achieved in image classification and object detection tasks. This is due to the fact that unrelated scene categories may share the same objects, and CNN lacks effective representation of objects that commonly appear within a scene. Unlike object or simple outdoor scenes (such as snow, grass, sky, etc.), indoor scenes often involve complex relationships between multiple objects, and the coexistence of objects between different scene categories often leads to feature similarity, making DNNs unable to achieve the accuracy of object recognition or outdoor scene recognition in the field of indoor scene recognition.

[0005] Therefore, existing research gives a scheme that combines the context of objects in a scene to recognize the scene, and obtains positive results by obtaining semantic information within the scene through semantic segmentation or object detection technology to assist scene recognition. Specifically, a method similar to statistics is used to reduce attention to non-discriminative objects during scene recognition according to the frequency of objects appearing in the scene. However, due to the complexity and diversity of indoor environments, it is quite difficult to select discriminative objects.

[0006] Or, semantic information is used to add semantic labels to the features in the backbone network, and the context relationship between feature regions with different semantic labels is explored to constrain scene recognition. For example, based on context information, in the SAS-Net model, the semantic features generated by the semantic segmentation score map are used to weight the feature maps generated by the RGB image at different positions, so that the network pays more attention to the discriminative regions in the scene image; the binary feature vector corresponding to the object in the scene is obtained through the detection network, and then combined with the backbone feature to assist the backbone network in indoor scene recognition; based on the ARG-Net model, the foreground and background regions in the scene are detected by the semantic segmentation technology, and the features obtained by the backbone network are combined to establish the context relationship (spatial relationship and morphological relationship) between the region features. At the same time, in order to fully integrate the semantic object information in the scene, based on the OTS-Net model, the semantic label map is used to provide position information for the feature representation obtained by the semantic segmentation downsampling network, so as to directly perform scene recognition on the segmentation network through the self-attention mechanism, but the diversity between the semantic segmentation task and the scene recognition task is ignored; Zhou et al. established the co-occurrence relationship of objects in the scene by using a probability method, and combined representative objects with the global representation of the scene to obtain a better scene representation, but the representation of the internal object information in the scene still remained on the surface.

[0007] The above methods try to establish the context relationship between semantic regions in the scene to guide scene classification, however, the accuracy of the above methods mainly depends on the feature extraction ability of the backbone network, and the object information only plays an auxiliary role, the utilization of internal object information is limited, resulting in poor accuracy of the final recognition. Moreover, due to the accuracy of the semantic segmentation or object detection technology, all methods that combine semantic information for scene recognition inevitably face a problem, that is, the negative impact of semantic ambiguity. The above methods do not deeply consider the problem of detection error or segmentation error when using object detection or semantic segmentation technology for scene recognition. To solve this problem, existing methods usually use a confidence threshold to filter and alleviate semantic ambiguity, but the threshold method is obviously not flexible enough, and its generalization ability is limited when facing huge data volume. SUMMARY

[0008] To solve the above problems of the prior art, the application provides an indoor scene recognition method and system based on a combined semantic region relationship model, constructs a semantic region relationship model (SRRM), fully mines the relationship between semantic regions of objects in an indoor scene, and effectively handles the semantic segmentation error problem in the model; meanwhile, in order to more comprehensively utilize the information contained in the scene, the SRRM is combined with PlacesCNN as a combined semantic region relationship model (CSRRM), a global representation is generated through feature combination, the complementary information between the two is more effectively mined, and thus the indoor scene recognition with better accuracy and generalization performance is realized.

[0009] In a first aspect, the application provides an indoor scene recognition method based on a combined semantic region relationship model.

[0010] An indoor scene recognition method based on a combined semantic region relationship model comprises the following steps.

[0011] An indoor scene image to be recognized is acquired.

[0012] The indoor scene image to be recognized is input into a combined semantic region relationship model, semantic global node features of the input image are extracted through a semantic region relationship module, global node features of the input image are extracted through an RGB feature extraction module, the semantic global node features and the global node features are aggregated through a global feature aggregation module to obtain global aggregated node features, the extracted global aggregated node features are input into a classifier, and a scene recognition result is output.

[0013] In a further technical solution, the semantic region relationship module comprises a semantic segmentation network and a semantic global node network; the extraction of the semantic global node features of the input image through the semantic region relationship module comprises the following steps.

[0014] The indoor scene image to be recognized is input into the semantic segmentation network to generate a semantic segmentation score map.

[0015] The acquired semantic segmentation score map is input into the semantic global node network to extract semantic global node features.

[0016] In a further technical solution, a confidence filtering layer is further arranged between the semantic segmentation network and the semantic global node network.

[0017] After the semantic segmentation score map is acquired, the semantic segmentation score map is filtered through the confidence filtering layer based on adaptive filtering, a filtered semantic segmentation score map is output, and the filtered semantic segmentation score map is input into the semantic global node network.

[0018] Further technical solutions, the semantic global node network comprises a ResBlock+ChAM module, wherein the ResBlock module comprises three Basic Blocks of the original network structure ResNet-50, and the ChAM module is added to the ResBlock module.

[0019] The feature map is extracted through the ResBlock module, and the extracted feature map is output to a channel attention map through the ChAM module; the channel attention map is multiplied with the input feature map element by element, and then added to the input feature map element by element, to output the semantic global node feature.

[0020] Further technical solutions, the ChAM module outputs the channel attention map, comprising:

[0021] The input feature map aggregates the spatial information of the feature map through average pooling operation and maximum pooling operation respectively, to obtain an average pooling feature vector and a maximum pooling feature vector respectively; the two feature vectors are added element by element after passing through a shared multi-layer perception, and the channel attention map is obtained and output after sigmoid activation.

[0022] Further technical solutions, the global feature aggregation module aggregates the semantic global node feature and the global node feature to obtain the global aggregated node feature, comprising:

[0023] The semantic global node feature and the global node feature are concatenated to obtain the concatenated feature;

[0024] The concatenated feature is output to a multi-layer perception based on a residual structure to obtain a multi-layer perception modified feature;

[0025] The modified feature is input into a DW convolution layer to obtain and output the global aggregated node feature.

[0026] In a second aspect, the present disclosure provides an indoor scene recognition system based on a combined semantic region relationship model.

[0027] An indoor scene recognition system based on a combined semantic region relationship model, comprising:

[0028] An image acquisition module is configured to acquire an indoor scene image to be recognized.

[0029] The indoor scene recognition module is configured to input an indoor scene image to be recognized into a combined semantic region relationship model, extract semantic global node features of the input image through a semantic region relationship module, extract global node features of the input image through an RGB feature extraction module, aggregate the semantic global node features and the global node features through a global feature aggregation module to obtain global aggregated node features, input the extracted global aggregated node features into a classifier, and output a scene recognition result.

[0030] Further, the global feature aggregation module is configured to aggregate the semantic global node features and the global node features to obtain the global aggregated node features.

[0031] The semantic global node features and the global node features are concatenated to obtain concatenated features.

[0032] The concatenated features are output through a multi-layer perception based on a residual structure to obtain modified features of the multi-layer perception.

[0033] The modified features are input into a DW convolution layer to obtain and output the global aggregated node features.

[0034] In a third aspect, the present disclosure further provides an electronic device, which comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, and when the computer instructions are run by the processor, the steps of the method in the first aspect are completed.

[0035] In a fourth aspect, the present disclosure further provides a computer readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the steps of the method in the first aspect are completed.

[0036] The above one or more technical solutions have the following beneficial effects:

[0037] 1. The present disclosure provides an indoor scene recognition method and system based on a combined semantic region relationship model, which constructs a semantic region relationship model, fully mines the relationship between semantic regions of objects in an indoor scene, and effectively handles semantic segmentation error problems in the model; combines SRRM and PlacesCNN as a combined semantic region relationship model, generates global representation through feature combination, more effectively mines complementary information between the two, more comprehensively utilizes information contained in the scene, and realizes more accurate and generalizable indoor scene recognition.

[0038] 2. The present disclosure provides a semantic region relationship model SRRM, which can directly establish semantic region relationships on semantic segmentation results, and is used for representation and recognition of indoor scenes, and an indoor scene recognition method based on the SRRM model is superior to all existing methods of using only semantic segmentation results for scene recognition.

[0039] 3、The present application proposes a simple but effective confidence filtering method, which can adaptively greatly improve the reliability of semantic segmentation results and reduce the negative impact of semantic ambiguity. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of these drawings are set to explain the application, and do not constitute an improper limitation on the application.

[0041] Figure 1 The overall architecture schematic diagram of the combined semantic region relationship model in the embodiment of the present application is shown in the figure;

[0042] Figure 2 The overall architecture schematic diagram of the semantic region relationship module in the embodiment of the present application is shown in the figure;

[0043] Figure 3 The schematic diagram of the confidence filtering process of a single channel in the embodiment of the present application is shown in the figure;

[0044] Figure 4 The architecture schematic diagram of the ResBlock+ChAM module in the embodiment of the present application is shown in the figure;

[0045] Figure 5 The overall architecture schematic diagram of the global feature aggregation module in the embodiment of the present application is shown in the figure;

[0046] Figure 6 The schematic diagram of the ablation study results of different object information data volumes in the embodiment of the present application is shown in the figure;

[0047] Figure 7 The architecture schematic diagram of different global feature aggregation modules in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0049] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0050] Due to the complex composition of indoor scenes, indoor scene recognition is still a challenging task in the field of computer vision, and how to fully utilize the semantic information within the scene has always been a key problem in indoor scene recognition. However, due to the accuracy of semantic segmentation, the existing methods of utilizing semantic information within the scene are still at the stage of auxiliary annotation or coexistence statistics, and few methods directly start from the semantic information within the scene to mine the context relationship between semantic information. Therefore, the present application proposes a semantic region relationship model (SRRM), which is different from the existing context-based method of assigning semantic labels to the backbone features. The present application designs a simple and effective network architecture for the semantic segmentation score map, which can directly explore the spatial context relationship between semantic objects within the scene on the semantic segmentation score map, and at the same time, it also brings better interpretability while making full use of semantic information. At the same time, in the face of the negative impact of semantic ambiguity, unlike the existing method which is limited to using a confidence threshold to deal with the problem hard, an adaptive confidence filtering method is proposed, which can adaptively filter the ambiguous points according to the state of the image itself, greatly improving the reliability of semantic segmentation. Finally, in order to make full use of the information contained in the scene, considering that the SRRM model taking the semantic segmentation map as the input does not consider the color and texture information in the image, therefore, the SRRM model is combined with the PlacesCNN taking the RGB picture as the input to form a combined semantic region relationship model (CSRRM), which can more effectively explore the complementary information between the two, optimize the indoor scene recognition, and improve the accuracy of indoor scene recognition.

[0051] Embodiment one

[0052] The embodiment provides a kind of indoor scene recognition method based on combined semantic region relationship model, comprising the following steps:

[0053] Obtain the indoor scene image to be identified;

[0054] The indoor scene image to be identified is input into combined semantic region relationship model, respectively through semantic region relationship module extraction input image semantic global node feature, through RGB feature extraction module extraction input image global node feature, again through global feature aggregation module semantic global node feature and global node feature are aggregated, obtain global aggregated node feature, the extracted global aggregated node feature is input into classifier, and the scene recognition result is output.

[0055] The key point of the method described in this embodiment is the construction of a combined semantic region relation model (CSRRM), which includes a semantic region relation module, an RGB feature extraction module, and a global feature aggregation module. The semantic region relation module includes the semantic region relation model (SRRM). Indoor scene recognition is performed using the SRRM to fully explore the relationships between semantic regions of objects within the indoor scene and effectively address semantic segmentation errors in the model.

[0056] The overall architecture of the semantic region relationship module is as follows: Figure 2 As shown, it includes a semantic segmentation network and a semantic global node network. It extracts semantic global node features from the input image by combining the semantic region relation module (SRRM). This includes: first, processing the indoor scene image to be recognized... The input is fed into a semantic segmentation network to generate a semantic segmentation score map. Then the obtained semantic segmentation score map The input is fed into a semantic global node network to extract semantic global node features. in, Let represent the semantic prediction probability distribution of the pixel at position (i,j) in image I, with a total of l semantic labels (l=150). The semantic segmentation network described above uses a VisionTransformer Adapter model pre-trained on a preset dataset, which uses weights pre-trained on the preset dataset. The semantic global node network outputs a set of semantic global node features. This feature is a high-level generalization of the semantic segmentation score map in both spatial and channel dimensions.

[0057] Based on the ResNet50 architecture, this is applied to the semantic segmentation score map. Based on specific channel characteristics, a simple and clear network architecture is constructed. Compared with the traditional ResNet50 architecture, the network proposed in this embodiment requires less computing power.

[0058] Furthermore, due to the limited accuracy of semantic segmentation networks, segmentation errors are often unavoidable. To reduce the negative impact of erroneous semantic labels, a confidence filtering layer is first used to filter the semantic segmentation score map. Considering the inflexibility and limited generalization ability of existing methods that use confidence thresholds to filter and alleviate semantic ambiguity, this embodiment does not rigidly rely on a certain threshold to filter semantically ambiguous points, but instead adopts an adaptive filtering method to flexibly filter each semantic segmentation score map.

[0059] In the embodiment, a confidence filtering layer is further arranged between the semantic segmentation network and the semantic global node network in the semantic region relationship module. After the semantic segmentation score map is obtained , the semantic segmentation score map is filtered through the confidence filtering layer based on adaptive filtering. Specifically, the confidence filtering layer uses a filter with a kernel size of 2*2 to smooth each channel of the score map. For each filtering domain that the filter passes through, the filter only retains the pixel with the highest confidence in the coverage range. After processing all channels of the semantic segmentation score map , the filtered semantic segmentation score map

[0060] The confidence filtering process of a single channel is shown in Figure 3 . Compared with the segmentation map M (the pixel value of each point of which represents the channel with the highest confidence in the range of 1*1), each pixel point in the segmentation map corresponding to M' represents the channel with the highest confidence in the 2*2*1 range in which it is located. Therefore, the above-mentioned method of the embodiment filters the semantic segmentation map by using the coverage range of the discriminative region instead of a threshold, and is adaptively adjusted according to the state of each image, which not only improves the accuracy but also has high universality. The confidence filtering not only improves the reliability of the semantic segmentation score map, but also reduces the input size of the convolutional neural network, thereby reducing the computing power consumption of the module.

[0061] After that, the filtered semantic segmentation score map is input into the semantic global node network to extract semantic global node features . When the filtered semantic segmentation score map is processed by convolution, considering that the pixel value of each channel in the score map represents the semantic prediction probability distribution of the pixel at (i,j) in the image I, a channel attention module (ChAM) is added in the convolution layer (not including 7*7conv) of the semantic global node network. The ChAM module can explore the relationship between different channels in the input feature map. Since the channels in the input score map represent the probability of the corresponding semantic class, ChAM is conducive to the network paying more attention to the key semantic classes in the picture by using the probability relationship.

[0062] In the embodiment, the semantic global node network is built on the basis of network architecture ResNET50, a channel attention module (ChAM) is added in the convolution layer, that is, the semantic global node network comprises a 7*7 convolution layer and a ResBlock+ChAM module, wherein the 7*7 convolution layer and the ResBlock module are both sub-modules of the network architecture ResNet50, in the ResNet50 architecture, the input image is first subjected to 7*7 convolution processing, and then the obtained feature is input into the ResBlock module for processing to extract the feature; the feature map is extracted through the ResBlock module The feature map outputs a channel attention map through the ChAM module. Specifically, the ResBlock+ChAM module is a ChAM module added in the ResBlock module, and the ResBlock module comprises three Basic Blocks (Basic Block 2, 3 and 4) of the original network structure ResNet-50, and a ChAM module is added between every two Basic Blocks, as shown in Figure 4 In the ChAM module, the input feature map is first aggregated by an average pooling operation AvgPool and a maximum pooling operation MaxPool to obtain feature vectors F avg and F max respectively, and then the two are elementarily added after being processed by a shared multi-layer perception MLP, and the channel attention map is obtained and output after being activated by a sigmoid function.

[0063] M c (F)=σ(MLP(F avg )+MLP(F max ))

[0064] =σ(W1(W0AvgPool(F)+W1(W0MaxPool(F)

[0065] wherein σ represents a sigmoid function, and represent the weights of the multi-layer perception MLP, ReLU activation function is contained in W0, and r=16 represents a reduction coefficient.

[0066] Then, the channel attention map output by the ChAM module is elementarily multiplied with the input feature map , and then elementarily added with the input feature map to output the semantic global node feature , which can be represented by the following formula:

[0067]

[0068] wherein F' represents semantic global node features F s , represents element-level multiplication, represents element-level addition.

[0069] In view of the fact that the semantic segmentation score map input to the SRRM lacks information such as color and texture of the input image, in order to fully exploit the information contained in the image, the embodiment constructs an RGB feature extraction module, which includes a PlacesCNN model, and combines the proposed SRRM and Places CNN model as a combined semantic region relationship model CSRRM.

[0070] As shown in Figure 1 , the proposed combined semantic region relationship model CSRRM contains two branches, one branch is an SRRM branch for scene recognition using a semantic segmentation score map, and the other branch is a PlacesCNN branch for scene recognition using an original RGB image.

[0071] The RGB feature extraction module basically adopts a ResNet50 architecture. Considering that the pooling layer will introduce translation invariance to the network, making the differences between different node features in the global node features output by the module become blurred, which is not conducive to the exploration of complementary relationships, therefore, the pooling layer in the ResNet50 architecture is deleted. The indoor scene image to be recognized is an RGB image) is input into the RGB feature extraction module, and global node features are output, wherein c = 2048. The above network model is obtained by pre-training based on a preset data set.

[0072] Then, a global feature aggregation module is constructed to explore the complementary information between the semantic global node features of the PlacesCNN branch and the global node features of the SRRM branch, and to obtain the aggregated features of the two Exploring the complementary information of the two global features is the key to scene recognition. In the embodiment, a shared multi-layer perceptron and a Depth Wise convolution are used to solve this problem. The overall architecture of the global feature aggregation module is as shown in Figure 5 .

[0073] First, the semantic global node features and the global node features are concatenated in each channel dimension (both and have a dimension of length c in the channel dimension) to obtain the concatenated features Then, the features F' are processed by a multi-layer perception (MLP) and a residual structure is used to alleviate overfitting, to obtain features which can be represented by the following formula:

[0074] F'' = F' + φ(W n (Drop(φ(W m F' + b m )) + b n )

[0075] wherein φ represents a GeLU activation function, and represent weights and biases of two fully connected layers inside the MLP, l = 8192, Drop represents a dropout regularization function with an inactivation rate of 0.1.

[0076] After the MLP modification process, the features F'' are input into a DW (Depth Wise) convolution layer, and two global node features are aggregated in the channel dimension to deeply explore the complementary information between them, to obtain global aggregated node features F o Each channel value of F can be represented by the following formula:

[0077]

[0078] wherein K is a DW convolution kernel with a size of 2x1xM, M = 2048; m th The filter is used to process the mth layer of channel in F'' to generate the mth layer of channel in F o .

[0079] Finally, the extracted global aggregated node features F o are input into a classifier, and a scene recognition result is output.

[0080] In the process of inputting the indoor scene image to be recognized into the combined semantic region relationship model for scene recognition, the combined semantic region relationship model is a trained network model, and the training process is as follows: a large number of indoor scene images with labeled scene recognition results are used as a training data set to train the constructed combined semantic region relationship model.

[0081] Since the RGB feature and the semantic feature have different focuses, and when training two branch networks at the same time, the RGB and semantic modification gravity centers of some indoor scene images are different, and the overall loss may hinder the optimization of the less distinctive feature module. In order to prevent the training process of the two branches from affecting each other, the embodiment adopts a two-stage training process: in the first stage, the RGB scene recognition branch and the semantic scene recognition branch are trained respectively (the former uses the Places365 pre-trained weight initialization, and the latter is trained from scratch); in the second stage, the weights of the RGB feature extraction module and the semantic region relationship module are frozen, and the global feature aggregation module is trained from scratch.

[0082] Further, in order to verify the superiority of the method described in the embodiment, the effectiveness of the method is evaluated on the publicly available indoor scene recognition dataset. The publicly available indoor scene recognition dataset includes MIT-67, reduced Places365, and reduced SUNRGB-D, etc. First, the experimental details are set, and the benchmark dataset is given, then the ablation experiment is carried out to determine the influence of each module on the proposed method, and then the proposed method and the existing technical solutions are compared. Specifically, the following aspects are included:

[0083] A. Experimental details

[0084] Semantic segmentation: the Vision Transformer Adapter model pre-trained on the ADE20K dataset is used as the semantic segmentation network model, and semantic segmentation is performed in the semantic region relationship module.

[0085] Hardware and software: the model architecture, training and evaluation process are all implemented using the Pytorch1.10.2 deep learning framework. The computer device running the model is equipped with an NVIDIA GeForce RTX3090 image processing unit and an AMD R9 5900x 12-core Processor CPU.

[0086] Hyperparameters: the Ali-G algorithm is used to optimize the trainable parameters in the network. Ali-G is an optimization algorithm for deep learning, and the performance of this optimizer is slightly lower than that of the SGD optimizer, but only the initial learning rate hyperparameter is needed, and there is no need to manually create a learning rate decay plan. In all experiments, the initial learning rate is set to 0.1, and in the second stage of training, to prevent overfitting, the momentum is set to 0.9, and the dropout regularization function is used in the final classifier, and the omission probability is 0.8. For all training processes, the random seed is set to 304 to ensure reproducibility.

[0087] B. Dataset

[0088] The MIT-67 dataset consists of 67 indoor scene categories, with a total of 15,620 images, and each scene category contains at least 100 images. According to the original evaluation scheme, there are 80 images for each category for training and 20 images for testing. Because of the large intra-class difference of indoor scenes, the evaluation of the MIT dataset is quite challenging.

[0089] The Places365 dataset is the largest and most challenging scene classification dataset to date, containing a wide range of indoor and outdoor scene categories. In this experiment, we use a simplified version of it, which only considers indoor scene categories. In order to make a fair comparison with other existing advanced indoor scene recognition methods, we use two scene category settings: Places365-7 and Places365-14. Places365-7 contains 7 indoor scenes: bathroom, bedroom, hallway, dining room, kitchen, living room, and office; Places365-14 contains 14 indoor scenes: balcony, bedroom, dining room, home office, kitchen, living room, stairs, bathroom, closet, garage, home theater, self-service laundry, game room, and bar.

[0090] The SUN RGB-D dataset is currently the largest RGB-D dataset, consisting of 3874 Microsoft Kinect v2 images, 3389 ASUS Xtion images, 2003 Microsoft Kinect v1 images, and 1159 Intel RealSense images. The diversity of categories and sources makes the SUN RGB-D dataset more suitable for verifying the generalization ability of algorithms, so we use the cropped SUN RGB-D dataset with the same categories as Places365-7, and test the model pre-trained on Place365-7 dataset on the SUN RGB-D dataset without retraining.

[0091] C. Ablation Study

[0092] By conducting an ablation study, the effectiveness of the method proposed in this embodiment is evaluated. Specifically, first, the impact of different SRRM architectures is evaluated; second, the impact of different ways of combining two global features is evaluated. In addition, the datasets used in the ablation experiment are all MIT-67 datasets.

[0093] 1) Comparison of semantic global node module architecture

[0094] In this part, only the semantic global node module is used for scene recognition, and the impact of network structure changes (confidence filtering, channel attention module) on the recognition effect is studied, and the experimental results are shown in Table 1 below.

[0095] Table 1 ablation results for different architectures of semantic global node module

[0096]

[0097] The results of Table 1 show that filtering the semantic segmentation score map with confidence can greatly improve the network performance. Compared with directly processing the original semantic segmentation score map, the filtered score map can make the network improve the recognition accuracy by 3.43% to 5.75%, and reduce the flops (floating point operations) by 17.89G. The channel attention module can make the network improve the recognition accuracy by 3.1% to 5.5%, and the flops (floating point operations) only increase by 0.1G.

[0098] At the same time, the influence of different confidence filtering domains on network recognition accuracy is also explored. It can be found that after adding the channel attention module to the network, the recognition accuracy produced by using a 2*2 size filtering domain to process the score map is 1.12% higher than that produced by using a 4*4 size filtering domain. This is because the confidence filtering operation can reduce the negative impact of semantic segmentation errors, but it can also cause the segmentation map to lose a certain amount of target information. Compared with a 2*2 size filtering domain, a 4*4 size filtering domain will cause the semantic segmentation score map to lose more target information, which shows that the size of the confidence filtering domain needs to be considered from both the loss of target information and the semantic segmentation error. But even if more target information is lost, compared with adding the original semantic segmentation score map to the network, a 4*4 size filtering domain still improves the recognition accuracy by 3.43%, which further illustrates the necessity of filtering semantic segmentation error points.

[0099] According to the above experiments, in the semantic global node module, a 2*2 filtering domain is first used to filter the score map with confidence, and then a ResNet50 architecture with a channel attention mechanism is used for subsequent processing.

[0100] 2) Influence of target information quantity

[0101] In order to evaluate the influence of target information on the semantic global node module, a group of ablation experiments were conducted to evaluate the influence of target quantity on indoor scene recognition on the MIT-67 and Places365-14 datasets, respectively. The results are as follows Figure 6As shown in FIG. 6, the horizontal axis represents the number of object information in the scene used by the semantic global node module, and its value range is 70 to 150 (the value interval is 10), and the required object information number is obtained in the order of the object list of ADE20K. Obviously, the accuracy of scene recognition is positively correlated with the number of target information, and the recognition accuracy is greatly improved with the increase of the number. For example, compared with using 70 semantic categories, using 150 semantic categories, the recognition accuracy of the former is 8.81% higher than that of the latter on the MIT-67 dataset, and the recognition accuracy of the former is 3.073% higher than that of the latter on the Places365-14 dataset.

[0102] At the same time, it can be found that with the increase of object categories, the recognition accuracy is not always increasing, sometimes the accuracy changes little or even decreases slightly, such as when the number of semantic categories increases from 80 to 90, on the Places365-14 dataset, the recognition accuracy of the former is 0.143% lower than that of the latter; on the MIT-67 dataset, the recognition accuracy of the former is also 0.448% lower than that of the latter. This is because the object categories that are increased at this time are objects such as cars, trucks, street lamps, towers, etc. that are irrelevant to indoor scenes. In this regard, the relevance of the semantic categories of the object model to scene recognition is also crucial to scene recognition.

[0103] 3) Combination method of two global node features

[0104] In this part, various ways are tried to aggregate global node features to verify the effectiveness of the proposed method, and two baselines are also compared, and the experimental results are shown in Table 2. In order to facilitate analysis, as shown in FIG. 7, the combination method of 2 is given. Figure 7

[0105] Table 2 shows the ablation results of different architectures of the global feature aggregation module

[0106]

[0107] The results of Table 2 show that using any combination method can produce better performance than the baseline method, which shows that it is effective to deeply explore the complementary information of semantic global features and RGB global features in the scene recognition task. Compared with the series method, the depth-wise convolution method not only consumes less computing resources, but also produces higher accuracy, which shows the advantage of the depth convolution method in exploring complementary information. Therefore, the embodiment finally selects the DW convolution that aggregates two global node features with the best performance.

[0108] D. Comparison with existing methods

[0109] ​In this section, the proposed method is compared with the state-of-the-art methods. The comparison is performed on four indoor scene datasets, i.e., MIT-67, Places365-7, Places365-14 and Reduced SUN RGB-D. When validating on Reduced SUN RGB-D, the model is only pre-trained on Place365-7 without re-training to verify the generalization performance of the model.

[0110] 1) Comparison of SRRM with existing methods

[0111] The SRRM is compared with the existing scene recognition methods that only use semantic segmentation or object detection to obtain semantic representation. The results are shown in Table 3 below. To make a fair comparison, the SRRM is pre-trained on the Places365 dataset to validate on MIT-67 and to perform a head- trained validation on the reduced Places365 dataset. The superior performance of the proposed SRRM shows that building a semantic region relationship model is effective for scene recognition.

[0112] Table 3 Comparison results of SRRM with existing methods

[0113]

[0114] 2) Comparison of CSRRM with existing methods

[0115] The proposed CSRRM is compared with the state-of-the-art methods on benchmark datasets, and the results are shown in Tables 4, 5 and 6, respectively.

[0116] It is clear that the proposed method greatly outperforms most of the existing indoor scene recognition methods in effectiveness and generality. Compared with other methods that use semantic information for scene recognition, the proposed method achieves better results, which shows that exploring advanced representation of semantic information is very effective and proves the feasibility of in-depth study of complementary information between RGB and semantic representation. In addition, the proposed CSRRM is also superior to the current multi-branch based methods, which shows that using semantic information as an additional information source and obtaining its advanced representation is effective. In summary, the above experiments prove the superiority and generalization performance of the proposed method in indoor scene recognition.

[0117] Table 4 Experimental results of different methods on dataset MIT-67

[0118]

[0119]

[0120] Table 5 Experimental results of different methods on dataset Places365-14

[0121]

[0122] Table 6 Experimental results of different methods on dataset Places365-7

[0123]

[0124] The embodiment aims to establish a semantic region relationship model for indoor scene recognition. First, a confidence filtering module is proposed to reduce the negative impact of semantic segmentation ambiguity, which provides a novel and effective idea for filtering semantic ambiguous points for all methods using semantic segmentation. Then, the SRRM model is proposed, which can directly use the results of semantic segmentation to establish a global representation of the semantic region relationship within the scene. Considering that the input of SRRM (semantic segmentation score map) lacks color and texture information, the SRRM model is combined with the PlacesCNN model with RGB input, and a global feature aggregation module is designed to deeply explore the complementary information between them, and to achieve better indoor scene recognition.

[0125] Embodiment two

[0126] The embodiment provides an indoor scene recognition system based on a combined semantic region relationship model, comprising:

[0127] An image acquisition module is configured to acquire an indoor scene image to be recognized.

[0128] An indoor scene recognition module is configured to input the indoor scene image to be recognized into the combined semantic region relationship model, extract semantic global node features of the input image through the semantic region relationship module, extract global node features of the input image through the RGB feature extraction module, aggregate the semantic global node features and the global node features through the global feature aggregation module to obtain global aggregated node features, input the extracted global aggregated node features into a classifier, and output a scene recognition result.

[0129] Embodiment three

[0130] The embodiment provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps in the indoor scene recognition method based on the combined semantic region relationship model are completed.

[0131] Embodiment four

[0132] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the indoor scene recognition method based on the combined semantic region relationship model.

[0133] The steps involved in the above embodiments two to four correspond to the method embodiment one, and the specific implementation can refer to the relevant description part of the embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying the instruction set for execution by the processor and causing the processor to perform any method in the present application.

[0134] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0135] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0136] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not used to limit the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. An indoor scene recognition method based on a combined semantic region relation model, characterized in that, include: Acquire images of the indoor scene to be identified; The indoor scene image to be identified is input into the combined semantic region relation model. The semantic global node features of the input image are extracted by the semantic region relation module and the global node features of the input image are extracted by the RGB feature extraction module. Then, the semantic global node features and global node features are aggregated through the global feature aggregation module to obtain the global aggregated node features, including: The semantic global node features and global node features are concatenated to obtain the concatenated features; The cascaded features are passed through a multilayer perceptron based on residual structure to output multilayer perceptron-modified features. The modified features are then input into the DW convolutional layer to obtain and output global aggregation node features. The extracted global aggregation node features are input into the classifier, and the scene recognition result is output. A confidence filtering layer is also provided between the semantic segmentation network and the semantic global node network. The confidence filtering layer uses a filter with a kernel size of 2×2 to smooth each channel of the score map. For each filtering domain it passes through, the filter only retains the pixel with the highest confidence within its coverage area.

2. The indoor scene recognition method based on a combined semantic region relationship model as described in claim 1, characterized in that, The step of extracting semantic global node features of the input image through the semantic region relation module includes: The indoor scene image to be identified is input into the semantic segmentation network to generate a semantic segmentation score map; The obtained semantic segmentation score map is input into the semantic global node network to extract semantic global node features.

3. The indoor scene recognition method based on a combined semantic region relationship model as described in claim 1, characterized in that, After obtaining the semantic segmentation score map, the semantic segmentation score map is filtered by a confidence filtering layer based on adaptive filtering, and the filtered semantic segmentation score map is output. The filtered semantic segmentation score map is then input into the semantic global node network.

4. The indoor scene recognition method based on a combined semantic region relationship model as described in claim 1, characterized in that, The semantic global node network includes a ResBlock+ChAM module, wherein the ResBlock module includes three Basic Blocks of the original network structure ResNet-50, and the ChAM module is added to the ResBlock module; The ResBlock module extracts feature maps, which are then passed through the ChAM module to output channel attention maps. The channel attention maps are multiplied element-wise with the input feature maps, and then added element-wise to the input feature maps to output semantic global node features.

5. The indoor scene recognition method based on a combined semantic region relationship model as described in claim 4, characterized in that, The output channel attention map via the ChAM module includes: The input feature map is aggregated with spatial information through average pooling and max pooling operations to obtain average pooling feature vectors and max pooling feature vectors, respectively. The two feature vectors are then passed through a shared multilayer perceptron and added element-wise. After sigmoid activation, the channel attention map is obtained and output.

6. An indoor scene recognition system based on a combined semantic region relation model, characterized in that, include: The image acquisition module is used to acquire images of the indoor scene to be identified. The indoor scene recognition module is used to input the indoor scene image to be recognized into the combined semantic region relation model. The semantic global node features of the input image are extracted by the semantic region relation module and the global node features of the input image are extracted by the RGB feature extraction module. Then, the semantic global node features and global node features are aggregated through the global feature aggregation module to obtain the global aggregated node features, including: The semantic global node features and global node features are concatenated to obtain the concatenated features; The cascaded features are passed through a multilayer perceptron based on residual structure to output multilayer perceptron-modified features; the modified features are then input into the DW convolutional layer to obtain and output global aggregation node features; The extracted global aggregation node features are input into the classifier, and the scene recognition result is output. A confidence filtering layer is also provided between the semantic segmentation network and the semantic global node network. The confidence filtering layer uses a filter with a kernel size of 2×2 to smooth each channel of the score map. For each filtering domain it passes through, the filter only retains the pixel with the highest confidence within its coverage area.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps of an indoor scene recognition method based on a combined semantic region relationship model as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of an indoor scene recognition method based on a combined semantic region relationship model as described in any one of claims 1-5.

Images