Indoor semantic image segmentation method based on knowledge distillation

Through a knowledge distillation method, combined with RGB-D image and spectrum analysis technology, the problem of large-scale annotation data dependence and training time-consuming in the existing technology is solved, and the accuracy and robustness of indoor semantic segmentation are improved. Achieve more efficient feature extraction and segmentation performance.

CN119068335APending Publication Date: 2024-12-03ENJOYOR COMPANY LIMITED
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411144578.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art has problems in indoor semantic segmentation that have strong dependence on large-scale annotation data, long training process and poor robustness, especially when dealing with complex and diverse indoor scenarios.

Method used

Using the indoor semantic image segmentation method based on knowledge distillation, the convolutional neural network is constructed, combined with RGB images and depth images for data augmentation and feature extraction, the full feature fusion is used and the pixel harmony module is used for feature fusion, and Qualcomm is introduced in the decoding stage. Fourier filters, combined with spectral deconvolution and Fourier technology for denoising, optimize model performance using knowledge distillation techniques at feature, relationship and response levels.

Benefits of technology

Improve the accuracy and robustness of indoor semantic segmentation, can capture structure and texture information in images more effectively, reduce intra-class variation, improve inter-class separability, reduce dependence on large-scale annotation data, and improve Segmentation performance and computing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068335A_ABST
    Figure CN119068335A_ABST
Patent Text Reader

Abstract

The invention discloses an indoor semantic image segmentation method based on knowledge distillation, and relates to the technical field of image processing. Comprising a training stage and a testing stage, a plurality of original indoor scene images, corresponding depth maps and real semantic segmentation images are selected to form a training set, a convolutional neural network is constructed, and the training set is input into the neural network for training; and taking the obtained weight vector and the bias item as an optimal weight vector and an optimal bias item of a convolutional neural network classification training model to obtain the convolutional neural network classification training model, inputting a to-be-detected original indoor scene image in a test set, and outputting an indoor semantic segmentation prediction map corresponding to the original scene image by a model network. According to the method, the segmentation precision, efficiency and inter-class separability are improved, and intra-class variation is effectively reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to an indoor semantic image segmentation method based on knowledge distillation. Background Art

[0002] The popularity of smart home and smart building technologies has led to a sharp increase in the demand for accurate perception and understanding of indoor environments. Smart home devices, such as smart speakers, smart lights, and home robots, need to accurately understand and interact with the indoor environment in order to provide a better user experience. For example, home robots need to identify different objects and areas in the room in order to efficiently perform tasks such as cleaning and delivering items. This puts higher demands on semantic segmentation technology, prompting its rapid development. Semantic segmentation has received widespread attention because the geometric information provided by depth data can supplement the visual information in RGB data. This data fusion significantly improves the accuracy and robustness of semantic segmentation tasks, especially in complex indoor environments.

[0003] Traditional indoor semantic segmentation methods mainly rely on hand-crafted features and classic machine learning algorithms. These methods usually involve extracting various features from images, such as color, texture, shape, and geometry, and then inputting these features into a classifier for pixel-level classification. Common feature extraction methods include scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), and local binary pattern (LBP). These methods can cope with the problems of illumination changes and posture changes to a certain extent by extracting local invariant features. However, hand-crafted features often have limitations and are difficult to fully capture the complex information in the image.

[0004] In terms of classifiers, support vector machines (SVM), random forests (RandomForest) and K-nearest neighbors (KNN) are widely used. These classic machine learning algorithms can handle data of a certain scale and achieve success in certain specific tasks. However, traditional methods perform poorly when dealing with complex and diverse indoor scenes. Due to problems such as lighting, occlusion and texture similarity in indoor environments, traditional methods are difficult to guarantee stable performance. In addition, these methods usually require a lot of manual adjustments and empirical parameter selection, resulting in poor robustness in practical applications.

[0005] With the rise of deep learning technology, the field of indoor semantic segmentation has ushered in revolutionary changes. The outstanding performance of convolutional neural networks (CNNs) in image processing tasks has made them the main tool for semantic segmentation. Fully convolutional networks (FCNs) are one of the earliest models that use CNNs for semantic segmentation. By replacing fully connected layers with convolutional layers, pixel-level classification of input images is achieved. The introduction of FCNs opened a new chapter in semantic segmentation technology and made deep learning methods rapidly popular in semantic segmentation tasks. Subsequently, models such as U-Net, SegNet, and DeepLab further improved the network structure and segmentation performance. For example, U-Net effectively combines features at different levels by introducing skip connections, thereby improving the accuracy of segmentation. Skip connections enable the network to better utilize low-level features and improve its ability to capture details. SegNet reduces computational complexity by designing an efficient decoder structure, allowing the network to have a faster inference speed while maintaining high performance. The DeepLab series of models introduces dilated convolutions and conditional random fields (CRFs), which improve the ability to process details of segmentation results while maintaining computational efficiency. However, although deep learning methods have demonstrated excellent performance in indoor semantic segmentation, they also have some disadvantages and challenges. First, deep learning models are highly dependent on large-scale annotated data. High-quality large-scale annotated datasets require a lot of time and resources to acquire and annotate, which can be a huge obstacle for many practical application scenarios. Second, the training process of deep learning models is usually very time-consuming and requires a lot of computing resources, especially when dealing with high-resolution images and complex network structures. This makes the cost of training deep learning models high and not suitable for all application scenarios.

[0006] Therefore, it is an urgent problem for those skilled in the art to propose an indoor semantic image segmentation method based on knowledge distillation to solve the difficulties existing in the prior art. Summary of the invention

[0007] In view of this, the present invention provides an indoor semantic image segmentation method based on knowledge distillation, which is used to solve the technical problems existing in the prior art.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for indoor semantic image segmentation based on knowledge distillation includes two processes: a training phase and a testing phase.

[0010] The training phase consists of the following steps:

[0011] S1-1, select multiple original indoor scene images and corresponding depth maps and real semantic segmentation images to form a training set;

[0012] S1-2, build a convolutional neural network;

[0013] S1-3, each original indoor scene image and the corresponding multi-spectral image in the training set are data enhanced as the original input image, and input into the convolutional neural network for training to obtain the corresponding semantic prediction image, denoted as

[0014] S1-4, calculating the loss function value between the semantic prediction image and the corresponding true semantic segmentation image;

[0015] S1-5, repeat S1-3 and S1-4 for V times in total until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained. The weight vector and bias term corresponding to the minimum function value are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, denoted as W best and b best , where V>1;

[0016] The testing phase includes the following steps:

[0017] S2-1, select the Pth group of indoor scene images to be tested in the test set and mark them as S p , where 1≤p≤P;

[0018] S2-2, input various channels of the Pth group of indoor scene images to be detected into the trained model, and the model network outputs the indoor semantic segmentation prediction map corresponding to the original scene image, denoted as

[0019] In the above method, optionally, the convolutional neural network in S1-2 is used for feature extraction and feature fusion;

[0020] The feature extraction includes an RGB image input layer, a depth image input layer, eight convolution blocks, three pixel harmony modules, and a high-pass Fourier filter module;

[0021] Feature fusion includes four full feature fuser modules.

[0022] In the above method, optionally, the convolutional neural network includes a teacher network and a student network, which are divided into an encoding stage and a decoding stage;

[0023] The coding phase includes:

[0024] The RGB image passes through the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block in sequence;

[0025] The depth image passes through the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block in sequence;

[0026] The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block correspond vertically to the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block;

[0027] The vertically corresponding convolution block outputs are respectively input into the first full feature fuser module, the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module;

[0028] The outputs of the first full feature fuser module, the second full feature fuser module, and the third full feature fuser module and the outputs of the second convolution block, the third convolution block, and the fourth convolution block are respectively input into the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module; the outputs of the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module are respectively input into the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module;

[0029] The fourth full feature fuser module is connected to the high pass Fourier filter module;

[0030] The decoding phase includes:

[0031] The high-pass Fourier filter module is connected to the ninth convolution block. The output of the ninth convolution block is added to the output of the third full feature fusion module at the pixel level and then input into the tenth convolution block. The output of the tenth convolution block is added to the output of the second full feature fusion module at the pixel level and then input into the eleventh convolution block. The output of the eleventh convolution block is added to the output of the first full feature fusion module at the pixel level and then input into the twelfth convolution block.

[0032] In the above method, optionally, the convolution block is composed of a plurality of Transformer blocks connected in sequence;

[0033] The Transformer block consists of an attention layer, a deep separation convolution DWConv, a skip connection, a DropPath, a normalization layer BatchNorm, an activation layer Act, and a multi-layer perceptron MLP connected in sequence.

[0034] In the above method, optionally, in the full feature fuser module, the output of the convolution block is added to the output of the pixel harmony module and then sequentially passes through a low-pass Fourier filter, a fifth maximum pooling, a nineteenth convolution block, a twentieth convolution block, and a third activation function. The output result of the third activation function is multiplied by the output of the convolution block and the pixel harmony module respectively and then added to obtain output six, and the output six is ​​multiplied by the output of the pixel harmony module and weight two to obtain a first multiplication result;

[0035] The output of the convolution block and the pixel harmony module output are respectively input into the multi-layer perceptron MLP through the maximum pooling layer and the average pooling layer. The output of the multi-layer perceptron MLP is multiplied by the pixel harmony module output and weight one to obtain the second multiplication result;

[0036] The first multiplication result, the second multiplication result and the convolution block output are added element by element to obtain the full feature fusion output.

[0037] In the above method, optionally, the pixel harmony module includes:

[0038] The input of the full feature fusion module passes through the 13th convolution block, the 14th convolution block, the 15th convolution block, and the 16th convolution block in sequence, and then performs a cascade operation to obtain output 2;

[0039] The input of the convolution module is sequentially subjected to the first maximum pooling, the seventeenth convolution block, the first activation function, the second maximum pooling, the eighteenth convolution block, and the second activation function to obtain output three and output four;

[0040] Input 2 is multiplied pixel-wise with output 3 and output 4 respectively and then added to obtain the pixel harmony module output.

[0041] In the above method, optionally, the convolutional neural network in S1-2 also includes knowledge distillation, and the knowledge distillation is divided into feature pixel matching distillation, relationship level distillation and response level distillation.

[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for indoor semantic image segmentation based on knowledge distillation, which has the following beneficial effects:

[0043] 1) The method of the present invention constructs a convolutional neural network, inputs the RGB image and the depth image into the convolutional neural network for training, and obtains a convolutional neural network classification training model; then the indoor scene image to be semantically segmented is input into the convolutional neural network classification training model to obtain the corresponding predicted semantic segmentation image. When constructing the convolutional neural network, the method of the present invention uses a full feature fuser to perform denoising using spectral deconvolution and Fourier technology. The pixel harmony module further refines the edge details by analyzing and reconstructing the pattern. A high-pass Fourier filter is used before the decoder to provide clearer input to ensure detailed and accurate multi-scale image segmentation.

[0044] 2) This invention can capture more structural and texture information by introducing frequency domain analysis technology, such as Fourier transform, to perform spectral decomposition of the image. Specifically, the RGB-D image is extracted through a convolutional neural network, and then these features are transformed in the frequency domain to obtain frequency domain features. Through techniques such as high-pass filtering and low-pass filtering, the details and overall structural information of the image are enhanced, and multi-level fusion is performed in the frequency domain and spatial domain. Such a method can not only capture the high-frequency details in the image, but also effectively extract the overall structural information of the low frequency, forming a more comprehensive feature representation.

[0045] 3) This invention uses three target knowledge distillation techniques to simplify model performance: Feature distillation performs hierarchical filtering on teacher and student features in the frequency domain during encoding to align their phases and amplitudes. Relation distillation constructs a spatial domain attention map and a pixel perception domain to highlight important feature pairs and domain-specific associations during decoding. Response distillation optimizes category-specific responses by integrating frequency and spatial domain features, effectively reducing intra-class variation and improving inter-class separability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A flow chart of an indoor semantic image segmentation method based on knowledge distillation provided by the present invention;

[0048] Figure 2 A schematic diagram of an indoor semantic image segmentation method based on knowledge distillation provided by the present invention;

[0049] Figure 3 This is a block diagram of the full-featured fuser module of the present invention;

[0050] Figure 4 This is a block diagram of the pixel harmony module implementation of the present invention;

[0051] Figure 5 This is a block diagram for implementing distillation learning in the present invention;

[0052] Figure 6a is the first original indoor scene image of the same scene;

[0053] Figure 6b It is the first original indoor scene GT picture of the same scene;

[0054] Figure 6c Using the method of the present invention Figure 6a The predicted image after distillation of the original indoor scene image shown;

[0055] Figure 7a is the second original indoor scene image of the same scene;

[0056] Figure 7b It is the second original indoor scene GT picture of the same scene;

[0057] Figure 7c Using the method of the present invention Figure 7a The predicted image after distillation of the original indoor scene image shown;

[0058] Figure 8a is the third original indoor scene image of the same scene;

[0059] Figure 8b This is the third original indoor scene GT image of the same scene;

[0060] Figure 8c Using the method of the present invention Figure 8a The predicted image after distillation of the original indoor scene image shown;

[0061] Figure 9a is the fourth original indoor scene image of the same scene;

[0062] Figure 9b This is the fourth original indoor scene GT image of the same scene;

[0063] Fig.9c Using the method of the present invention Figure 9a The predicted image after distillation of the original indoor scene image is shown. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] See also Figure 1 As shown, the present invention discloses an indoor semantic image segmentation method based on knowledge distillation, including two processes: a training phase and a testing phase; wherein,

[0066] The training phase consists of the following steps:

[0067] S1-1, select multiple original indoor scene images and corresponding depth maps and real semantic segmentation images to form a training set;

[0068] S1-2, build a convolutional neural network;

[0069] S1-3, each original indoor scene image and the corresponding multi-spectral image in the training set are data enhanced as the original input image, and input into the convolutional neural network for training to obtain the corresponding semantic prediction image, denoted as

[0070] S1-4, calculating the loss function value between the semantic prediction image and the corresponding true semantic segmentation image;

[0071] S1-5, repeat S1-3 and S1-4 for V times in total until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained. The weight vector and bias term corresponding to the minimum function value are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, denoted as W best and b best , where V>1;

[0072] The testing phase includes the following steps:

[0073] S2-1, select the Pth group of indoor scene images to be tested in the test set and mark them as S p , where 1≤p≤P;

[0074] S2-2, input various channels of the Pth group of indoor scene images to be detected into the trained model, and the model network outputs the indoor semantic segmentation prediction map corresponding to the original scene image, denoted as

[0075] Furthermore, the training set constructed in S1-1 is specifically to record the qth original indoor scene image as {S q (i,j)}, and the training set is q (i,j)} corresponding to the real semantic segmentation image is recorded as Among them, the indoor scene image is an RGB color image, Q is a positive integer, such as Q = 795, q is a positive integer, 1≤q≤Q, Represents the pixel value of the pixel with coordinate position (i, j) in {S(i, j)}. The original indoor scene images are mainly composed of RGB images and multispectral images taken in different environments. RGB images record the spectral information of three bands: red, green, and blue, while multispectral images capture the spectral information of other different bands. The information of each band is equivalent to an independent channel component, that is, each original scene image containing a salient object includes the R channel, G channel, and B channel components of the RGB image and the corresponding depth image (Depth).

[0076] Furthermore, in S1-2, convolutional neural networks are used for feature extraction and feature fusion;

[0077] The feature extraction includes an RGB image input layer, a depth image input layer, eight convolution blocks, three pixel harmony modules, and a high-pass Fourier filter module;

[0078] Feature fusion includes four full feature fuser modules.

[0079] Furthermore, the high-pass Fourier filter is specifically operated as follows: First, initialize a tensor with the same shape as the input feature to store the filtered result. Then traverse each channel number of the feature and convert it into a numpy array and normalize it. Perform a two-dimensional Fourier transform on the processed image and move the center of the spectrum to the center of the image. Then generate a high-pass filter, set the center to 1, the rest of the area to 0, and apply the filter for frequency domain filtering. After filtering, perform an inverse Fourier transform back to the spatial domain, then convert the array into a tensor and copy it to the channel corresponding to the result feature map.

[0080] Further, refer to Figure 2 As shown, the convolutional neural network includes a teacher network and a student network, which are divided into an encoding stage and a decoding stage;

[0081] The coding phase includes:

[0082] The RGB image passes through the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block in sequence;

[0083] The depth image passes through the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block in sequence;

[0084] The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block correspond vertically to the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block;

[0085] The vertically corresponding convolution block outputs are respectively input into the first full feature fuser module, the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module;

[0086] The outputs of the first full feature fuser module, the second full feature fuser module, and the third full feature fuser module and the outputs of the second convolution block, the third convolution block, and the fourth convolution block are respectively input into the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module; the outputs of the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module are respectively input into the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module;

[0087] The fourth full feature fuser module is connected to the high pass Fourier filter module;

[0088] The decoding phase includes:

[0089] The high-pass Fourier filter module is connected to the ninth convolution block. The output of the ninth convolution block is added to the output of the third full feature fusion module at the pixel level and then input into the tenth convolution block. The output of the tenth convolution block is added to the output of the second full feature fusion module at the pixel level and then input into the eleventh convolution block. The output of the eleventh convolution block is added to the output of the first full feature fusion module at the pixel level and then input into the twelfth convolution block.

[0090] Specifically, the input of the convolutional neural network is the RGB image and depth image of each picture (the width and height of the picture are divided into W = 480, H = 640, the RGB channels are R channel components, G channel components and B channel components, and the depth map also has three channels) through the Segformer-b4 and Segformer-b2 backbone networks to extract teacher features and student features. The network models of the teacher and the student are the same, and the structures of the two feature extraction flows are similar.

[0091] Furthermore, the convolutional block is composed of multiple Transformer blocks connected in sequence;

[0092] The Transformer block consists of an attention layer, a deep separation convolution DWConv, a skip connection, a DropPath, a normalization layer BatchNorm, an activation layer Act, and a multi-layer perceptron MLP connected in sequence.

[0093] Specifically, the attention layer in the convolution block is linearly transformed by Query, Key, and Value (the input and output dimensions are 64); the depthwise separable convolution has a kernel size of 3×3, a stride of 1, an edge padding of 1, a head number of 1, and a dropout of 0; the first fully connected layer in the MLP layer passes through the GELU activation function and then enters the DWConv layer, with a kernel size of 3×3, a stride of 1, and an edge padding of 1, and then passes through the second fully connected layer, and again applies the skip connection and DropPath (value is 0) operations before being normalized and output.

[0094] The first convolutional block and the fifth convolutional block are composed of three Transformer blocks connected in sequence. The scaling ratio of each attention layer is 1 / 64. The first fully connected layer in the MLP layer has an input dimension of 64 and an output dimension of 256. The second fully connected layer has an input dimension of 256 and an output dimension of 64. The feature map sets output by the first and fifth convolutional blocks are recorded as The width of the feature map is Height is

[0095] The second convolutional block and the sixth convolutional block structure are composed of four Transformer blocks connected in sequence. Each Transformer block contains two attention heads, and the scaling ratio of each attention layer is changed to 1 / 128. The first fully connected layer in the MLP layer has an input dimension of 128 and an output dimension of 512. The second fully connected layer has an input dimension of 512 and an output dimension of 128. The feature map sets output by the second and seventh convolutional blocks are recorded as The width of the feature map is Height is

[0096] The third convolutional block and the seventh convolutional block structure are composed of four Transformer blocks connected in sequence. Each Transformer block contains five attention heads, and the scaling ratio of each attention layer becomes 1 / 320. The first fully connected layer in the MLP layer has an input dimension of 320 and an output dimension of 1280. The second fully connected layer has an input dimension of 1280 and an output dimension of 320. Each DropPath is 0.1. The feature map sets output by the third and seventh convolutional blocks are recorded as The width of the feature map is Height is

[0097] The fourth convolutional block and the eighth convolutional block are composed of three Transformer blocks connected in sequence. Each Transformer block contains 8 attention heads, and the scaling ratio of each attention layer becomes 1 / 512. The first fully connected layer in the MLP layer has an input dimension of 512 and an output dimension of 2048. The second fully connected layer has an input dimension of 2048 and an output dimension of 512. Each DropPath is 0.1. The feature map sets output by the fourth and eighth convolutional blocks are recorded as The width of the feature map is Height is

[0098] The Segformer-b4 used by the teacher model differs from the Segformer-b2 used by the student model in terms of depth, i.e. the number of attention heads is 3, 8, 27, and 3.

[0099] Further, see Figure 3 As shown, in the full feature fuser module, the output of the convolution block is added to the output of the pixel harmony module and then passes through the low-pass Fourier filter, the fifth maximum pooling, the nineteenth convolution block, the twentieth convolution block, and the third activation function in sequence. The output result of the third activation function is multiplied by the output of the convolution block and the pixel harmony module respectively and then added to obtain output six. The output six is ​​multiplied by the pixel harmony module output and the weight two to obtain the first multiplication result.

[0100] The output of the convolution block and the pixel harmony module output are respectively input into the multi-layer perceptron MLP through the maximum pooling layer and the average pooling layer. The output of the multi-layer perceptron MLP is multiplied by the pixel harmony module output and weight one to obtain the second multiplication result;

[0101] The first multiplication result, the second multiplication result and the convolution block output are added element by element to obtain the full feature fusion output.

[0102] Specifically, the specific operation of the low-pass Fourier filter in the full feature fuser module is as follows: first, the input tensor is converted into a grayscale image, then the grayscale image is subjected to a two-dimensional Fourier transform and converted to the frequency domain, the number of rows and columns of the image is obtained to calculate the center position of the image, a low-pass filter is selected (the center position is set to 0, and the others are set to 1), and the mask generated by frequency domain filtering is multiplied element by element with the image, and the result is inverse Fourier transformed back to the spatial domain.

[0103] Further, see Figure 4 As shown, the Pixel Harmony module includes:

[0104] The input of the full feature fusion module passes through the 13th convolution block, the 14th convolution block, the 15th convolution block, and the 16th convolution block in sequence, and then performs a cascade operation to obtain output 2;

[0105] The input of the convolution module is sequentially subjected to the first maximum pooling, the seventeenth convolution block, the first activation function, the second maximum pooling, the eighteenth convolution block, and the second activation function to obtain output three and output four;

[0106] Input 2 is multiplied pixel-wise with output 3 and output 4 respectively and then added to obtain the pixel harmony module output.

[0107] Specifically, the 13th and 16th convolution blocks have 1×3 convolution kernels, a step size of 2, and are padded to (0, 1); the 14th and 15th convolution blocks have 3×1 convolution kernels, a step size of 2, and are padded to (1, 0). The activation function is the sigmoid activation function.

[0108] Furthermore, in the decoding stage, the ninth convolution block contains a basic convolution block (convolution, normalization, activation operation), where the convolution kernel is 3, the stride is 2, the padding is 1, and a transposed convolution, where the convolution kernel is 2 and the stride is 2. The eleventh and twelfth convolution blocks contain two basic convolution blocks, a dropout operation (value is 0.5), and a transposed convolution.

[0109] Further, see Figure 5 As shown, the convolutional neural network in S1-2 also includes knowledge distillation, which is divided into feature pixel matching distillation, relationship level distillation, and response level distillation.

[0110] Furthermore, taking the specific teacher network of feature pixel matching distillation as an example, the output data of the first teacher encoder and the output data of the second teacher encoder are input into the first hierarchical filter, and the filtering scale of the first hierarchical filter is 3; the output data of the second teacher encoder and the output data of the third teacher encoder are input into the second hierarchical filter, and the filtering scale of the second hierarchical filter is 5; the output data of the third teacher encoder and the output data of the fourth teacher encoder are input into the third hierarchical filter, and the filtering scale of the third hierarchical filter is 7. The first hierarchical filter, the second hierarchical filter and the third hierarchical filter are connected in parallel with the twenty-first convolution block, and the output of the twenty-first convolution block is successively adjusted by the fourth activation function, the twenty-second convolution block, and the fifth activation function to obtain the feature-level outputs of the four teacher networks. The operation of the student network is similar to that of the teacher network. The only difference is that the output after activation function seven continues to perform pixel-level multiplication with the twenty-fourth convolution block. In order to narrow the distance with the teacher features, it is added with the teacher features corresponding to each layer and then output. The corresponding student output Output with teachers Perform pixel matching distillation to improve the accuracy of model prediction, and the loss is

[0111] Further, relational distillation uses the attention transfer mapping method for distillation. First, adjust the size of the first layer output of the decoder to match the second layer and then add them. The addition results are respectively input to the 26th convolution block, the 27th convolution block, and the 28th convolution block for information search and adjustment to obtain the corresponding output. The three outputs are added again and respectively input to the first channel averaging and the first channel maximization to obtain a more comprehensive feature expression. The first channel averaging output and the first channel maximization output are added and input to the 29th convolution block. Then, a weight is obtained through the ninth activation function and multiplied by the output after multi-scale convolution to obtain the final feature. Finally, the attention transfer loss function is used to calculate the similarity between the teacher and the student, and the loss is

[0112] Furthermore, response-level distillation first calculates the teacher's final prediction result and the student's prediction probability distribution for each pixel. and Then, the softmax function adjusted with temperature T is used to traverse each pixel of each batch and calculate the output probability distribution of the two models at each position. Finally, the Kullback-Leibler divergence between the predicted distributions of the teacher and student models at each position is calculated, and the loss is

[0113] Furthermore, the data enhancement in S1-3 includes performing data enhancement on each original indoor scene image and the corresponding multi-spectral image in the training set by adjusting brightness, contrast, color, and randomly flipping and cropping.

[0114] Furthermore, the loss function value in S1-4 is calculated using cross-entropy loss to calculate the true semantic segmentation image. A collection of semantic segmentation prediction maps The loss function value between

[0115] Furthermore, the number of training times in S1-5 is selected as 200 times.

[0116] Furthermore, there are 654 groups of indoor scene images to be detected in S2-1.

[0117] In a specific embodiment, the architecture of the RGB-D indoor semantic segmentation network based on the knowledge distillation collaborative multi-signal injection network is built using the Python-based deep learning library Pytorch3.6. The test sets of the NYUDv2 and SUN-RGBD databases are used to analyze the indoor semantic segmentation effect using the method of the present invention. We use three commonly used objective parameters for evaluating semantic segmentation methods as evaluation indicators, class accuracy (CA), pixel accuracy (PA), and the ratio of the intersection and union of the segmented image and the labeled image (MIoU) to evaluate the segmentation performance of the predicted semantic segmentation image.

[0118] The method of the present invention is used to predict each indoor image in the indoor semantic database NYUDv2 and the SUN-RGBD test set, and the predicted semantic segmentation image corresponding to each indoor scene image is obtained. The category accuracy CA, pixel accuracy PA, and the ratio of the intersection and union of the segmented image and the label image MIoU reflecting the semantic segmentation effect of the method of the present invention are listed in Table 1. From the data listed in Table 1, it can be seen that the segmentation result of the indoor scene image obtained by the method of the present invention is good, indicating that it is feasible and effective to use the method of the present invention to obtain the predicted semantic segmentation image corresponding to the indoor scene image.

[0119] Table 1 Evaluation results of the method of the present invention on the test set

[0120] method PA.(%) CA.(%) mIoU.(%) Params(M) Flops(G) Ours 77.9 69.9 54.8 30.9 97

[0121] Figure 6a The first original indoor scene image of the same scene is given; Figure 6b The first original indoor scene GT image of the same scene is given; Figure 6c The method of the present invention is given Figure 6a The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 7a The second original indoor scene image of the same scene is given; Figure 7b The second original indoor scene GT image of the same scene is given; Figure 7c The method of the present invention is given Figure 7a The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 8a The third original indoor scene image of the same scene is given; Figure 8b The third original indoor scene GT image of the same scene is given; Figure 8c The method of the present invention is given Figure 8a The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 9a The fourth original indoor scene image of the same scene is given; Figure 9b The fourth original indoor scene GT image of the same scene is given; Fig.9c The method of the present invention is given Figure 9a The original indoor scene image shown in Figure 1 is used for prediction to obtain the predicted semantic segmentation image. Figure 6a , Figure 6b and Figure 6c ,contrast Figure 7a , Figure 7b and Figure 7c ,contrast Figure 7a , Figure 8b and Figure 8c ,contrast Figure 9a , Figure 9b and Fig.9c It can be seen that the segmentation accuracy of the predicted semantic segmentation image obtained by the method of the present invention is high.

[0122] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0123] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for indoor semantic image segmentation based on knowledge distillation, characterized in that: It includes two processes: training phase and testing phase; The training phase consists of the following steps: S1-1, select multiple original indoor scene images and corresponding depth maps and real semantic segmentation images to form a training set; S1-2, build a convolutional neural network; S1-3, each original indoor scene image and the corresponding multi-spectral image in the training set are data enhanced as the original input image, and input into the convolutional neural network for training to obtain the corresponding semantic prediction image, denoted as S1-4, calculating the loss function value between the semantic prediction image and the corresponding true semantic segmentation image; S1-5, repeat S1-3 and S1-4 for V times in total until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained. The weight vector and bias term corresponding to the minimum function value are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, denoted as W best and b best , where V>1; The testing phase includes the following steps: S2-1, select the Pth group of indoor scene images to be tested in the test set and mark them as S p , where 1≤p≤P; S2-2, input various channels of the Pth group of indoor scene images to be detected into the trained model, and the model network outputs the indoor semantic segmentation prediction map corresponding to the original scene image, denoted as 2. The indoor semantic image segmentation method based on knowledge distillation according to claim 1, characterized in that: In S1-2, convolutional neural networks are used for feature extraction and feature fusion; The feature extraction includes an RGB image input layer, a depth image input layer, eight convolution blocks, three pixel harmony modules, and a high-pass Fourier filter module; Feature fusion includes four full feature fuser modules.

3. The indoor semantic image segmentation method based on knowledge distillation according to claim 2, characterized in that: The convolutional neural network includes a teacher network and a student network, which are divided into an encoding stage and a decoding stage; The coding phase includes: The RGB image passes through the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block in sequence; The depth image passes through the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block in sequence; The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block correspond vertically to the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block; The vertically corresponding convolution block outputs are respectively input into the first full feature fuser module, the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module; The outputs of the first full feature fuser module, the second full feature fuser module, and the third full feature fuser module and the outputs of the second convolution block, the third convolution block, and the fourth convolution block are respectively input into the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module; the outputs of the first pixel harmony module, the second pixel harmony module, and the third pixel harmony module are respectively input into the second full feature fuser module, the third full feature fuser module, and the fourth full feature fuser module; The fourth full feature fuser module is connected to the high pass Fourier filter module; The decoding phase includes: The high-pass Fourier filter module is connected to the ninth convolution block. The output of the ninth convolution block is added to the output of the third full feature fusion module at the pixel level and then input into the tenth convolution block. The output of the tenth convolution block is added to the output of the second full feature fusion module at the pixel level and then input into the eleventh convolution block. The output of the eleventh convolution block is added to the output of the first full feature fusion module at the pixel level and then input into the twelfth convolution block.

4. The indoor semantic image segmentation method based on knowledge distillation according to claim 3 is characterized in that: The convolution block is composed of multiple Transformer blocks connected in sequence; The Transformer block consists of an attention layer, a deep separation convolution DWConv, a skip connection, a DropPath, a normalization layer BatchNorm, an activation layer Act, and a multi-layer perceptron MLP connected in sequence.

5. The indoor semantic image segmentation method based on knowledge distillation according to claim 3, characterized in that: In the full feature fuser module, the output of the convolution block is added to the output of the pixel harmony module and then passes through the low-pass Fourier filter, the fifth maximum pooling, the nineteenth convolution block, the twentieth convolution block, and the third activation function in sequence. The output result of the third activation function is multiplied by the output of the convolution block and the pixel harmony module respectively and then added to obtain output six. Output six is ​​multiplied by the output of the pixel harmony module and weight two to obtain the first multiplication result. The output of the convolution block and the pixel harmony module output are respectively input into the multi-layer perceptron MLP through the maximum pooling layer and the average pooling layer. The output of the multi-layer perceptron MLP is multiplied by the pixel harmony module output and weight one to obtain the second multiplication result; The first multiplication result, the second multiplication result and the convolution block output are added element by element to obtain the full feature fusion output.

6. The indoor semantic image segmentation method based on knowledge distillation according to claim 3, characterized in that: Pixel Harmony modules include: The input of the full feature fusion module passes through the 13th convolution block, the 14th convolution block, the 15th convolution block, and the 16th convolution block in sequence, and then performs a cascade operation to obtain output 2; The input of the convolution module is sequentially subjected to the first maximum pooling, the seventeenth convolution block, the first activation function, the second maximum pooling, the eighteenth convolution block, and the second activation function to obtain output three and output four; Input 2 is multiplied pixel-wise with output 3 and output 4 respectively and then added to obtain the pixel harmony module output.

7. The indoor semantic image segmentation method based on knowledge distillation according to claim 1, characterized in that: The convolutional neural network in S1-2 also includes knowledge distillation, which is divided into feature pixel matching distillation, relationship level distillation, and response level distillation.