Home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation

Abstract

The invention relates to the technical field of computer visual recognition, in particular to a home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation, which specifically comprises the following steps: establishing a teacher model taking SegformerB4 as a framework and a student model taking DeepLabV3 as a framework, and respectively extracting global context features and local detail features; enabling the student model to obtain the deep global heterogeneous features of the teacher model through a re-blocking cross attention feature alignment module, and making up the local receptive field defect of the convolutional network; a heterogeneous scale feature alignment module is adopted to reconstruct multi-scale feature maps output by different networks, and the problem of resolution mismatching caused by structural difference is eliminated; through a dual feature alignment mechanism, the student model fully learns knowledge representation of the teacher model, and the segmentation precision is improved while the size of the student model is kept unchanged. Through the process, the semantic knowledge of the teacher model is migrated to the student model, so that a lightweight but high-performance segmentation network is trained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision recognition technology, and in particular to a home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation. Background Art

[0002] Segmentation is one of the four major image recognition tasks in computer vision, including semantic segmentation, scene segmentation, and instance segmentation. It aims to determine which object or category each pixel belongs to. Semantic segmentation divides the entire image into multiple regions, each assigned to a specific category. Regions without a category are assumed to be background. Semantic segmentation technology is crucial in smart homes. It uses in-depth analysis of images or videos within the home environment to accurately identify and classify different objects and regions. This fine-grained environmental perception provides critical data for smart home systems, enabling them to more accurately respond to user needs and environmental changes. For example, by identifying different areas and objects in the home, smart home systems can intelligently manage energy consumption, adjusting heating or air conditioning systems based on usage and demand in different areas, improving energy efficiency. By identifying unusual behavior or the presence of strangers, such as intruders or unexpected situations, smart home systems can promptly issue alerts and take appropriate measures, enhancing the safety of the user's home environment.

[0003] Currently, the technology for deploying lightweight models based on convolutional neural network architectures on smart home devices is relatively mature. However, due to the inductive bias of convolutional neural networks, these models face difficulties in modeling long-range dependencies. This limitation leads to inaccurate scene understanding in smart homes, which in turn affects the intelligence level of smart home systems. In recent years, various vision transformer (VIT) semantic segmentation models based on the Transformer architecture have surpassed the segmentation performance of most models based on convolutional neural network architectures. However, models based on the Transformer architecture are computationally more complex, have slower training and inference speeds, and use a large amount of memory, making them difficult to deploy on resource-constrained smart home devices. To improve the accuracy of smart home systems' understanding of the environment while balancing the need for model deployment on devices with limited computing resources, we introduce a knowledge distillation method.

[0004] Knowledge distillation is a model compression method. The trained model can extract information from the input data, and then obtain the knowledge that plays a key role in the final task from the information. These learned knowledge is fixed in the form of parameters in the model, and when we input data into the trained model, we can obtain the knowledge formed by the model about the data. Knowledge distillation migrates the knowledge of the original model (teacher model) to the compressed model (student model), so that the compressed model can also learn similar knowledge although it is much smaller than the original model. However, the existing semantic segmentation knowledge distillation method mainly focuses on homogeneous model distillation, and lacks distillation design for heterogeneous models (based on the Transformer architecture and the convolutional neural network architecture). Therefore, we focus on designing a heterogeneous feature alignment module. Specifically, we propose a strategy of first blocking and then aligning the features generated by the convolutional layer to improve the problem of performance degradation caused by the mismatch of heterogeneous features and direct distillation. At the same time, in view of the problem of size mismatch of heterogeneous features, we introduce offset calculation, and first resample the size-mismatched heterogeneous features based on the offset, and then distill them, to improve the problem of performance degradation caused by direct downsampling or convolution matching feature size.

[0005] Therefore, the present application proposes a household semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation to solve the above problems. SUMMARY

[0006] The present application is aimed at the deficiencies of the prior art, and develops a household semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation. The present application can migrate the semantic knowledge of the teacher model to the student model through the reblocking cross-attention feature alignment module PCAM and the heterogeneous scale feature alignment module HSAM, so as to train a lightweight but high-performance segmentation network.

[0007] A household semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation, comprising the following steps: S1, collect household scene image data sets, preprocess the household scene images in the data set, and then divide the data set into a training set, a validation set and a test set; S2, construct a teacher model and a student model for household scene visual segmentation, construct a teacher model based on a semantic segmentation model based on the Transformer architecture and a Segformer-B4 segmentation network, and construct a student model based on a semantic segmentation model based on the convolutional neural network architecture and a DeepLabV3 model with a residual network ResNet-18 as a feature extraction backbone network; S3. Input the images in the training set into the teacher model and the student model for segmentation. During this period, the features extracted by the teacher model and the student model in the encoding stage are aligned through the re-block cross attention feature alignment module PCAM and the heterogeneous scale feature alignment module HSAM, respectively, and then the re-block cross attention feature alignment loss is obtained respectively. and heterogeneous scale feature alignment loss ; S4. Calculate the total loss of the student model and optimize the student model to obtain the optimized student model. The total loss of the student model includes the re-block cross-attention feature alignment loss. , heterogeneous scale feature alignment loss and the task loss of the student model ; S5. Input the images in the validation set into the fully trained and optimized student model for evaluation and verification, adjust the performance indicators of the student model, and obtain the best student model; input the data in the test set into the best student model to test it.

[0008] S1 is as follows: Home scene images are extracted from the online gallery. After manual review, images that do not meet the requirements are deleted. The remaining images form a dataset of home scene images. Then, data augmentation is performed on the images in the dataset. The data augmentation processing operations include setting the image cropping size, randomly scaling the images, and randomly flipping the images horizontally. Finally, the data-augmented images are proportionally divided into training set, validation set, and test set, which are used for training, validation, and testing of the student model, respectively. Among them, the images in the training set and validation set are annotated with the manual segmentation results.

[0009] S2 is as follows: S2.1. Build a teacher model for visual segmentation of home scenes: The teacher model includes an encoder and a decoder; The teacher model's encoder consists of four Transformer feature encoding blocks: the first, second, third, and fourth Transformer feature encoding blocks. Each Transformer feature encoding block contains multiple self-attention operations and feedforward neural network layers. The teacher model's decoder uses a full multi-layer perceptron (MLP) architecture. S2.2. Constructing a student model for visual segmentation of home scenes: The student model includes an encoder and a decoder. The encoder of the student model includes four residual convolution blocks and an ASPP void spatial pyramid pooling module. The four residual convolution blocks are the first residual convolution block, the second residual convolution block, the third residual convolution block and the fourth residual convolution block. The decoder of the student model includes a Bilinear bilinear interpolation module, a convolution module and a softmax normalization module.

[0010] The images in the training set are input into the teacher model and the student model for segmentation. The specific process is as follows; In the teacher model, the image passes through four Transformer feature encoding blocks in sequence. The first Transformer feature encoding block outputs the embedded feature , embedded features Input to the second Transformer feature encoding block output embedding features , embedded features Input to the third Transformer feature encoding block output embedding features , embedded features Input to the fourth Transformer feature encoding block output embedding features ; Then embed the features 、 、 and Input to the full multi-layer perceptron MLP architecture, output the segmented image predicted by the teacher model; In the student model, the image passes through four residual convolution blocks in sequence, and the first residual convolution block outputs the embedded feature ,feature Input to the second residual convolution block output feature ,feature Input to the third residual convolution block output feature ,feature Input to the fourth residual convolution block output feature ; Then the feature Input to the ASPP void space pyramid pooling module and output features , the features The input is sent to the Bilinear interpolation module for upsampling to restore its spatial details; finally, the number of channels of the feature map is mapped to the number of categories through the convolution module, and each pixel is classified through the Softmax normalization module to generate the segmentation image predicted by the student model .

[0011] The operations in the re-blocked cross-attention feature alignment module PCAM are as follows: The features output by the student model 、 and , the embedding features output by the teacher model , and , are divided into blocks in the form of feature maps, and then mapped into the form of block embeddings, to obtain features , and , the number and size of which are consistent with the embedding features output by the teacher model. The calculation formula of the above operation is as follows: , wherein, represents the feature output by the residual convolution block, represents the embedding feature , represents the size of the block of the embedding feature , represents the step of the block of the embedding feature represents the embedding feature output by the i th Transformer feature encoding block, represents the feature after the i th block, represents the block operation, represents the convolution mapping operation. Then, the features are respectively input into the teacher query linear layer and the teacher key linear layer in the PCAM module, to obtain the teacher block embedding query vector and the teacher block embedding key vector ; the feature is input into the student query linear layer in the PCAM module, to obtain the student block embedding query vector . The calculation formula of the above operation is as follows: , , , , Next, the transpose of the teacher block embedding query vector is multiplied by the teacher block embedding key vector , and the numerical processing is performed through the normalized exponential function , to obtain the self-attention map of the teacher model ; at the same time, the transpose of the student block embedding query vector is multiplied by the teacher block embedding key vector , and the numerical processing is performed through the normalized exponential function .​Perform numerical processing to obtain the cross attention map of the student-teacher model , the calculation formula for the above operation is as follows: , , Finally, the self-attention map of the teacher model and the cross attention map of the student-teacher model By maximizing the cosine Perform similarity alignment, weighted summation and finally obtain the re-block cross attention feature alignment loss , the calculation formula is as follows: .

[0012] The operations in the heterogeneous scale feature alignment module HSAM are as follows: Embed the features output by the third Transformer feature encoding block in the teacher model , according to the features output by the third residual convolution block in the student model In the form of Mapped from block embedding to feature map form, and then interpolated to obtain features , so that the feature Resolution and characteristics The calculation formula is as follows: , in, Represents the operation of remapping from block embedding form to feature map form, Representation characteristics The batch size, Representation characteristics height, Representation characteristics The width, Represents an interpolation operation; The features and features Input to the two residual convolution modules in the HSAM module respectively and Perform feature extraction and obtain features and features Then the two features are fused to obtain the feature , the calculation formula is as follows: , , , in, represents the student residual convolution block, represents the teacher residual convolution block, Represents element-wise bitwise multiplication; Then the fused features Input into the two convolution modules in the HSAM module respectively and , and obtain the features and features Then, the two features are spliced ​​along the channel to obtain the feature , the calculation formula is as follows: , , , Among them, the convolution module and They are all 3×3 convolutional layers with a stride and padding of 1. represents the student convolution module, represents the teacher convolutional module, Represents a splicing operation; Then the spliced ​​features Input them into the two offset generation modules in the HSAM module respectively to obtain the offset map of the teacher model and the offset map of the student model , the calculation formula is as follows: , , in, represents the student offset generation module, represents the teacher offset generation module; Then based on the offset map of the teacher model and the offset map of the student model By reconstructing the function and features Reconstruction, the calculation formula is as follows: , , in, Represents the feature The features obtained after reconstruction are Represents the feature The features obtained after reconstruction are Represents a reconstruction operation; The features to be reconstructed and By maximizing the cosine Similarity alignment to obtain heterogeneous scale feature alignment loss , the specific calculation formula is as follows: .

[0013] The reconstruction operations are as follows: , in, represents the input of the offset operation, represents the input feature map, Represents relative to the feature map The offset map, The pixel position is The reconstruction operation of the input feature map is 、Width , offset map At each pixel position there is a corresponding height component and width component, and Represents the offset map exist The height and width components at Indicates pixel position, pixel position and All are feature maps Pixels in Indicates the current pixel After offset and traversed pixels The distance relationship in the height direction, Indicates the current pixel After offset and traversed pixels The distance relationship in the width direction, Indicates taking the maximum value.

[0014] The total loss of the student model is calculated as follows: , in, represents the task loss of the student model, which is calculated as follows: , in, represents the cross entropy loss function, represents the segmentation image predicted by the student model, and Represents images respectively The height and width, Represents the predicted segmentation image At pixel position The predicted output of Represents manually segmented images At pixel position The true label.

[0015] S5 is as follows: The image data in the validation set is fed into the fully trained and optimized student model for validation evaluation. The performance indicators on unseen validation data are calculated. The training process is then monitored to determine whether the model is overfitting or underfitting. Hyperparameters are then tuned to select the best student model. The images in the test set are input into the best student model to run the test and make a final evaluation of the performance of the best student model.

[0016] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects: The present invention discloses a home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation. A teacher model based on the SegformerB4 (based on the attention mechanism) architecture and a student model based on the DeepLabV3 (based on the convolutional network) architecture are established to extract global context features and local detail features, respectively. In view of the differences between the teacher and student models, the present invention maps the features extracted by the student model into blocks, and then uses a patched cross-attention alignment module (PCAM) to enable the student model to obtain the deep global heterogeneous features of the teacher model, thereby compensating for the local receptive field defects of the convolutional network. A heterogeneous scale alignment module (HSAM) is used to reconstruct the multi-scale feature maps output by different networks to eliminate the resolution mismatch problem caused by structural differences. The dual feature alignment mechanism enables the student model to fully learn the knowledge representation of the teacher model, thereby improving segmentation accuracy while keeping the size of the student model unchanged. Unlike existing distillation methods, this method introduces an offset value generated based on the fused features before aligning heterogeneous features of different scales to reconstruct the heterogeneous features of different scales, avoiding the performance degradation caused by simply making the resolution of heterogeneous features consistent through downsampling or convolution. By constructing two different alignment methods, the present invention enables the student model to fully learn the deep feature information obtained by the teacher model and eliminate structural differences, thereby achieving the purpose of model compression while improving the segmentation accuracy of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0018] Figure 1 Schematic diagram of the method structure of the present invention.

[0019] Figure 2 Schematic diagram of the internal structure of the re-block cross-attention feature alignment module in the present invention.

[0020] Figure 3 Schematic diagram of the internal structure of the heterogeneous scale feature alignment module in the present invention. DETAILED DESCRIPTION

[0021] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0022] Example 1 like Figures 1 to 3 As shown in Figure 1, a home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation is shown in Figure 1. The specific steps are as follows: S1. Collect home scene images to form a dataset, preprocess the home scene images in the dataset, and then divide the dataset into a training set, a validation set, and a test set; S2. Construct a teacher model and student model for visual segmentation of home scenes. The teacher model is based on a semantic segmentation model with a Transformer architecture and a Segformer-B4 segmentation network. The student model is based on a semantic segmentation model with a convolutional neural network architecture and a DeepLabV3 model with a ResNet-18 residual network as the feature extraction backbone network. S3. Input the images in the training set into the teacher model and the student model for segmentation. During this period, the features extracted by the teacher model and the student model in the encoding stage are aligned through the re-block cross attention feature alignment module PCAM and the heterogeneous scale feature alignment module HSAM, respectively, and then the re-block cross attention feature alignment loss is obtained respectively. and heterogeneous scale feature alignment loss ; S4. Calculate the total loss of the student model and optimize the student model to obtain the optimized student model. The total loss of the student model includes the re-block cross-attention feature alignment loss. , heterogeneous scale feature alignment loss and the task loss of the student model ; S5. Input the images in the validation set into the fully trained and optimized student model for evaluation and verification, adjust the performance indicators of the student model, and obtain the best student model; input the data in the test set into the best student model to test it.

[0023] The Segformer segmentation network has different settings of B1, B2, B3, B4 and B5, the main difference lies in the number of times of encoder self-attention operation, the number of block embedding, etc., wherein the model parameter quantity of Segformer-B4 is large, and the segmentation performance is high, therefore, the teacher model is constructed by selecting Segformer-B4.

[0024] In the specific implementation, S1 is specifically as follows: The home scene images are extracted from the network gallery, the images not meeting the requirements are deleted through manual review, the remaining images form a data set of home scene images, then the images in the data set are subjected to data enhancement, the data enhancement processing operation includes setting the image cropping size, randomly scaling the image and randomly horizontally flipping the image, finally, the images subjected to data enhancement are divided into a training set, a verification set and a test set in proportion, which are respectively used for training, verification and testing of the student model, wherein the images in the training set and the verification set have annotations of artificial segmentation results.

[0025] In the specific implementation, S2 is specifically as follows: S2.1, constructing a teacher model for home scene visual segmentation: The teacher model includes an encoder and a decoder; The encoder of the teacher model includes four Transformer feature encoding blocks, which are a first Transformer feature encoding block, a second Transformer feature encoding block, a third Transformer feature encoding block and a fourth Transformer feature encoding block, each Transformer feature encoding block contains multiple self-attention operations and feedforward neural network layers, and the decoder of the teacher model adopts a full multi-layer perception MLP architecture; The full multi-layer perception MLP architecture includes an MLP Layer multi-layer perception layer and an MLP multi-layer perception, the MLP Layer multi-layer perception layer includes a radial change layer and a Relu activation function, and the MLP multi-layer perception includes an input layer, multiple MLP Layer multi-layer perception layers and an output layer; the multi-scale features from the encoder are fused through the full multi-layer perception MLP architecture, specifically, the feature maps of different resolutions are transformed in the channel dimension, then the feature maps are restored to the same size as the input image through the up-sampling operation, and finally the segmentation image predicted by the teacher model is generated.

[0026] S2.2, constructing a student model for home scene visual segmentation: The student model comprises an encoder and a decoder, the encoder of the student model comprises four residual convolutional blocks and an ASPP empty space pyramid pooling module, the four residual convolutional blocks are respectively a first residual convolutional block, a second residual convolutional block, a third residual convolutional block and a fourth residual convolutional block, and the decoder of the student model comprises a Bilinear interpolation module, a convolutional module and a softmax normalization module.

[0027] In the embodiment, the images in the training set are respectively input into the teacher model and the student model for segmentation, and the specific process is as follows: In the teacher model, the image sequentially passes through four Transformer feature encoding blocks, the first Transformer feature encoding block outputs embedded features , the embedded features are input into the second Transformer feature encoding block to output embedded features , the embedded features are input into the third Transformer feature encoding block to output embedded features , the embedded features are input into the fourth Transformer feature encoding block to output embedded features ; then the embedded features , , and are input into a full multi-layer perceptron (MLP) architecture to output a segmentation image predicted by the teacher model. In the student model, the image sequentially passes through four residual convolutional blocks, the first residual convolutional block outputs embedded features , the features are input into the second residual convolutional block to output features , the features are input into the third residual convolutional block to output features , the features are input into the fourth residual convolutional block to output features ; then the features are input into an ASPP empty space pyramid pooling module to output features , the features are input into a Bilinear interpolation module for upsampling to restore spatial details; finally, the convolutional module is used to map the channel number of the feature map to the class number, and the softmax normalization module is used to classify each pixel to generate a segmentation image predicted by the student model .

[0028] In the embodiment, the operation in the reblocking cross-attention feature alignment module (PCAM) is as follows: The features output by the student model 、 and , according to the embedding features output by the teacher model 、 and In the form of feature maps, the feature maps are divided into blocks and then mapped into block embedding to obtain the feature 、 and , the number and size of its blocks are consistent with the embedding features output by the teacher model. The calculation formula for the above operation is as follows: , ; in, Indicates the The features output by the residual convolution block, Represents embedded features The size of the chunk, Represents embedded features The step size of the block, Indicates the The embedded features output by the Transformer feature encoding block, Indicates the After the block features, Indicates block operation, Represents the convolution mapping operation; Then, the features Input into the teacher query linear layer in the PCAM module respectively and the teacher key linear layer , get the query vector embedded in the teacher block and the key vector of the teacher block embedding ; The feature Input to the student query linear layer of the PCAM module , get the student block embedding query vector , the calculation formula for the above operation is as follows: , , , Next, the query vector embedded in the teacher block The transpose of and the key vector of the teacher block embedding Multiply and pass the normalized exponential function Perform numerical processing to obtain the self-attention map of the teacher model ; Also embed the student block into the query vector The transpose of and the key vector of the teacher block embedding Multiply and pass the normalized exponential function Perform numerical processing to obtain the cross attention map of the student-teacher model , the calculation formula for the above operation is as follows: , , Finally, the self-attention map of the teacher model and the cross attention map of the student-teacher model By maximizing the cosine Perform similarity alignment, weighted summation and finally obtain the re-block cross attention feature alignment loss , the calculation formula is as follows: .

[0029] In a specific implementation, the operations in the heterogeneous scale feature alignment module HSAM are as follows: Embed the features output by the third Transformer feature encoding block in the teacher model , according to the features output by the third residual convolution block in the student model In the form of Mapped from block embedding to feature map form, and then interpolated to obtain features , so that the feature Resolution and characteristics The calculation formula is as follows: , in, Represents the operation of remapping from block embedding form to feature map form, Representation characteristics The batch size, Representation characteristics height, Representation characteristics The width, Represents the interpolation operation, which can change the resolution of the feature map so that the resolution of the processed feature map meets the requirements; The features and features Input to the two residual convolution modules in the HSAM module respectively and Perform feature extraction and obtain features and features Then the two features are fused to obtain the feature , the calculation formula is as follows: , , , wherein, represents a student residual convolution block, represents a teacher residual convolution block, represents an element-wise multiplication, the fused feature is further input into two convolution modules in the HSAM module, and respectively to obtain features and respectively, and then the two features are spliced along the channel to obtain a feature , and the calculation formula is as follows: , , , wherein, the convolution modules and are both 3x3 convolution layers with a step of 1 and a padding of 1, represents a student convolution module, represents a teacher convolution module, represents a splicing operation, which can splice the features along the channel dimension; then the spliced feature is input into two offset generation modules in the HSAM module to obtain a teacher model offset map and a student model offset map , and the calculation formula is as follows: , , wherein, represents a student offset generation module, represents a teacher offset generation module; then, based on the teacher model offset map and the student model offset map , the features and are reconstructed by a reconstruction function, and the calculation formula is as follows: , , wherein, represents a feature obtained by reconstructing the feature , represents a feature obtained by reconstructing the feature ,​ Represents a reconstruction operation; The features to be reconstructed and By maximizing the cosine Similarity alignment to obtain heterogeneous scale feature alignment loss , the specific calculation formula is as follows: .

[0030] In a specific implementation, the reconstruction operation is as follows: , in, represents the input of the offset operation, represents the input feature map, Represents relative to the feature map The offset map, The pixel position is The reconstruction operation of the input feature map is 、Width , offset map At each pixel position there is a corresponding height component and width component, and Represents the offset map exist The height and width components at Indicates pixel position, pixel position and All are feature maps Pixels in Indicates the current pixel After offset and traversed pixels The distance relationship in the height direction, Indicates the current pixel After offset and traversed pixels The distance relationship in the width direction, Indicates taking the maximum value.

[0031] In a specific implementation, the total loss calculation process of the student model is as follows: , in, represents the task loss of the student model, which is calculated as follows: , in, represents the cross entropy loss function, represents the segmentation image predicted by the student model, and Represents images respectively height and width of the image, denotes a predicted segmentation image at pixel position predicted output, denotes a manually segmented image at pixel position true label.

[0032] In the detailed implementation, S5 is specifically as follows: The validation set and the test set are strictly kept independent in the whole model training and knowledge distillation process, and do not participate in any form of weight update or optimization step, which is a basic principle to ensure the objectivity of model evaluation and the reliability of generalization ability; The image data in the validation set is input into the student model after being fully trained and optimized for validation evaluation, and the performance indicators of the student model on unseen validation data are calculated, so as to monitor the training process to judge whether the model is overfitting or underfitting, and to select the best student model through hyperparameter optimization; The image in the test set is input into the best student model for running test, and the final evaluation of the real generalization performance of the best student model on completely independent and new data is carried out, so as to judge whether the optimized student model meets the expected requirements.

[0033] Embodiment 2 In order to better prove the technical effect of the present application, the method in the present application is compared with existing methods in different situations, and the existing methods include SKD knowledge distillation method (the model is derived from IEEE Conference on Computer Vision and Pattern Recognition, CVPR “Structured Knowledge Distillation for Semantic Segmentation”, CVPR 2019.), IFVD intra-class feature variation distillation method (the model is derived from IEEE / CVF, European Conference on Computer Vision, ECCV “Intra-class Feature Variation Distillation for Semantic Segmentation”, ECCV 2020.) and CWD channel-level knowledge distillation method (the model is derived from Recognition, CVPR “Channel-wise Knowledge Distillation for Dense Prediction”, CVPR 2020.); The model used for the experiment is the Householdscenes dataset constructed by collecting household scene images by oneself, 10384 household scene pictures are crawled from the network, 122 pictures that do not meet the requirements are excluded after manual review, and the remaining 10262 pictures form the household scene image dataset Householdscenes. The dataset has a total of 8000 training pictures as the training set, 1000 pictures with public annotations for verification as the verification set, and 1262 pictures with annotations for benchmarking as the test set. After the image collection is completed, data enhancement processing is performed: set the picture cropping size to 512*512, randomly scale the picture according to a ratio of 0.9, and randomly flip the picture horizontally according to a probability of 50%.

[0034] As shown in Table 1, the segmentation performance of the method in the application and the existing method on the Householdscenes dataset is compared.

[0035] Table 1 Comparison of segmentation performance of the method in the application and the existing method on the Householdscenes dataset In Table 1, T represents the teacher model constructed in the application, S represents the student model constructed in the application, S+SKD represents the student model S constructed in the application selecting the SKD knowledge distillation method, S+IFVD represents the student model S constructed in the application selecting the IFVD intra-class feature variation distillation method, S+CWD represents the student model S constructed in the application selecting the CWD channel-level knowledge distillation method, S+PCAM represents the student model S constructed in the application selecting the PCAM reblocking cross-attention feature alignment module in the application for knowledge distillation, S+HSAM represents the student model S constructed in the application selecting the HSAM heterogeneous scale feature alignment module in the application for knowledge distillation, and S+PCAM+HSAM represents the student model S constructed in the application selecting the PCAM and HSAM in the application for knowledge distillation. The mean intersection over union mIoU is a core indicator for evaluating semantic segmentation. The higher the value, the higher the spatial overlap accuracy between the predicted result and the true annotation. As shown in Table 1, when the student model S does not use any method, the average intersection over union is the lowest. After using the existing method and the method of the application for knowledge distillation, the average intersection over union is improved. Among them, the average intersection over union is the highest when the PCAM and HSAM provided by the application are selected for knowledge distillation. Therefore, the student model selects the PCAM and HSAM provided by the application for knowledge distillation, and the segmentation accuracy is higher. In addition, it can be seen from Table 1 that the size of the teacher model is 61.4MB and the size of the student model is 12.68MB. Although the average intersection-over-union ratio of the student model using PCAM and HSAM for knowledge distillation in the present invention is lower than that of the teacher model, the size of the student model is much smaller than that of the teacher model. This shows that the method in the present invention not only improves the segmentation accuracy of the student model, but also achieves the purpose of model compression.

[0036] Although the above describes the specific implementation methods of the invention in conjunction with the accompanying drawings, it does not limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A home semantic segmentation model compression method based on multi-layer feature alignment knowledge distillation, characterized by: The following steps are involved: S1. Collect home scene images to form a dataset, preprocess the home scene images in the dataset, and then divide the dataset into a training set, a validation set, and a test set; S2. Construct a teacher model and student model for visual segmentation of home scenes. The teacher model is based on a semantic segmentation model with a Transformer architecture and a Segformer-B4 segmentation network. The student model is based on a semantic segmentation model with a convolutional neural network architecture and a DeepLabV3 model with a ResNet-18 residual network as the feature extraction backbone network. S3. Input the images in the training set into the teacher model and the student model for segmentation. During this period, the features extracted by the teacher model and the student model in the encoding stage are aligned through the re-block cross attention feature alignment module PCAM and the heterogeneous scale feature alignment module HSAM, respectively, and then the re-block cross attention feature alignment loss is obtained respectively. and heterogeneous scale feature alignment loss ; S4. Calculate the total loss of the student model and optimize the student model to obtain the optimized student model. The total loss of the student model includes the re-block cross-attention feature alignment loss. , heterogeneous scale feature alignment loss and the task loss of the student model ; S5. Input the images in the validation set into the fully trained and optimized student model for evaluation and verification, adjust the performance indicators of the student model, and obtain the best student model; input the data in the test set into the best student model to test it.

2. The method for compressing a home semantic segmentation model based on multi-layer feature alignment knowledge distillation according to claim 1 is characterized in that: S1 is as follows: Home scene images are extracted from the online gallery. After manual review, images that do not meet the requirements are deleted. The remaining images form a dataset of home scene images. Then, data augmentation is performed on the images in the dataset. The data augmentation processing operations include setting the image cropping size, randomly scaling the images, and randomly flipping the images horizontally. Finally, the data-augmented images are proportionally divided into training set, validation set, and test set, which are used for training, validation, and testing of the student model, respectively. Among them, the images in the training set and validation set are annotated with the manual segmentation results.

3. The method for compressing a home semantic segmentation model based on multi-layer feature alignment knowledge distillation according to claim 2 is characterized in that: S2 is as follows: S2.

1. Build a teacher model for visual segmentation of home scenes: The teacher model includes an encoder and a decoder; The teacher model's encoder consists of four Transformer feature encoding blocks: the first, second, third, and fourth Transformer feature encoding blocks. Each Transformer feature encoding block contains multiple self-attention operations and feedforward neural network layers. The teacher model's decoder uses a full multi-layer perceptron (MLP) architecture. S2.

2. Constructing a student model for visual segmentation of home scenes: The student model includes an encoder and a decoder. The encoder of the student model includes four residual convolution blocks and an ASPP void spatial pyramid pooling module. The four residual convolution blocks are the first residual convolution block, the second residual convolution block, the third residual convolution block and the fourth residual convolution block. The decoder of the student model includes a Bilinear bilinear interpolation module, a convolution module and a softmax normalization module.

4. The method for compressing a home semantic segmentation model based on multi-layer feature alignment knowledge distillation according to claim 3 is characterized in that: The images in the training set are input into the teacher model and the student model for segmentation. The specific process is as follows; In the teacher model, the image passes through four Transformer feature encoding blocks in sequence. The first Transformer feature encoding block outputs the embedded feature , embedded features Input to the second Transformer feature encoding block output embedding features , embedded features Input to the third Transformer feature encoding block output embedding features , embedded features Input to the fourth Transformer feature encoding block output embedding features ; Then embed the features 、 、 and Input to the full multi-layer perceptron MLP architecture, output the segmented image predicted by the teacher model; In the student model, the image passes through four residual convolution blocks in sequence, and the first residual convolution block outputs the embedded feature ,feature Input to the second residual convolution block output feature ,feature Input to the third residual convolution block output feature ,feature Input to the fourth residual convolution block output feature ; Then the feature Input to the ASPP void space pyramid pooling module and output features , the features The input is sent to the Bilinear interpolation module for upsampling to restore its spatial details; finally, the number of channels of the feature map is mapped to the number of categories through the convolution module, and each pixel is classified through the Softmax normalization module to generate the segmentation image predicted by the student model .

5. The method for compressing a home semantic segmentation model based on multi-layer feature alignment and knowledge distillation according to claim 4 is characterized in that: The operations in the re-blocked cross-attention feature alignment module PCAM are as follows: The features output by the student model 、 and , according to the embedding features output by the teacher model 、 and In the form of feature maps, the feature maps are divided into blocks and then mapped into block embedding to obtain the feature 、 and , the number and size of its blocks are consistent with the embedding features output by the teacher model. The calculation formula for the above operation is as follows: ， ； in, Indicates the The features output by the residual convolution block, Represents embedded features The size of the chunk, Represents embedded features The step size of the block, Indicates the The embedded features output by the Transformer feature encoding block, Indicates the After the block features, Indicates block operation, Represents the convolution mapping operation; Then, the features Input into the teacher query linear layer in the PCAM module respectively and the teacher key linear layer , get the query vector embedded in the teacher block and the key vector of the teacher block embedding ; The feature Input to the student query linear layer of the PCAM module , get the student block embedding query vector , the calculation formula for the above operation is as follows: ， ， ， Next, the query vector embedded in the teacher block The transpose of and the key vector of the teacher block embedding Multiply and pass the normalized exponential function Perform numerical processing to obtain the self-attention map of the teacher model ; Also embed the student block into the query vector The transpose of and the key vector of the teacher block embedding Multiply and pass the normalized exponential function Perform numerical processing to obtain the cross attention map of the student-teacher model , the calculation formula for the above operation is as follows: ， ， Finally, the self-attention map of the teacher model and the cross attention map of the student-teacher model By maximizing the cosine Perform similarity alignment, weighted summation and finally obtain the re-block cross attention feature alignment loss , the calculation formula is as follows: 。 6. The method for compressing a home semantic segmentation model based on multi-layer feature alignment knowledge distillation according to claim 5 is characterized in that: The operations in the heterogeneous scale feature alignment module HSAM are as follows: Embed the features output by the third Transformer feature encoding block in the teacher model , according to the features output by the third residual convolution block in the student model In the form of Mapped from block embedding to feature map form, and then interpolated to obtain features , so that the feature Resolution and characteristics The calculation formula is as follows: ， in, Represents the operation of remapping from block embedding form to feature map form, Representation characteristics The batch size, Representation characteristics height, Representation characteristics The width, Represents an interpolation operation; The features and features Input to the two residual convolution modules in the HSAM module respectively and Perform feature extraction and obtain features and features Then the two features are fused to obtain the feature , the calculation formula is as follows: ， ， ， in, represents the student residual convolution block, represents the teacher residual convolution block, Represents element-wise bitwise multiplication; Then the fused features Input into the two convolution modules in the HSAM module respectively and , and obtain the features and features Then, the two features are spliced ​​along the channel to obtain the feature , the calculation formula is as follows: ， ， ， Among them, the convolution module and They are all 3×3 convolutional layers with a stride and padding of 1. represents the student convolution module, represents the teacher convolutional module, Represents a splicing operation; Then the spliced ​​features Input them into the two offset generation modules in the HSAM module respectively to obtain the offset map of the teacher model and the offset map of the student model , the calculation formula is as follows: ， ， in, represents the student offset generation module, represents the teacher offset generation module; Then based on the offset map of the teacher model and the offset map of the student model By reconstructing the function and features Reconstruction, the calculation formula is as follows: ， ， in, Represents the feature The features obtained after reconstruction are Represents the feature The features obtained after reconstruction are Represents a reconstruction operation; The features to be reconstructed and By maximizing the cosine Similarity alignment to obtain heterogeneous scale feature alignment loss , the specific calculation formula is as follows: 。 7. The method for compressing a home semantic segmentation model based on multi-layer feature alignment and knowledge distillation according to claim 6 is characterized in that: The reconstruction operations are as follows: ， in, represents the input of the offset operation, represents the input feature map, Represents relative to the feature map The offset map, The pixel position is The reconstruction operation of the input feature map is 、Width , offset map At each pixel position there is a corresponding height component and width component, and Represents the offset map exist The height and width components at Indicates pixel position, pixel position and All are feature maps Pixels in Indicates the current pixel After offset and traversed pixels The distance relationship in the height direction, Indicates the current pixel After offset and traversed pixels The distance relationship in the width direction, Indicates taking the maximum value.

8. The method for compressing a home semantic segmentation model based on multi-layer feature alignment and knowledge distillation according to claim 7 is characterized in that: The total loss of the student model is calculated as follows: ， in, represents the task loss of the student model, which is calculated as follows: ， in, represents the cross entropy loss function, represents the segmentation image predicted by the student model, and Represents images respectively The height and width, Represents the predicted segmentation image At pixel position The predicted output of Represents manually segmented images At pixel position The true label.

9. The method for compressing a home semantic segmentation model based on multi-layer feature alignment and knowledge distillation according to claim 8 is characterized in that: S5 is as follows: The image data in the validation set is fed into the fully trained and optimized student model for validation evaluation. The performance indicators on unseen validation data are calculated. The training process is then monitored to determine whether the model is overfitting or underfitting. Hyperparameters are then tuned to select the best student model. The images in the test set are input into the best student model to run the test and make a final evaluation of the performance of the best student model.

Images