Image processing method based on feature pyramid and knowledge guided knowledge distillation

Abstract

The application discloses an image processing method based on feature pyramid and knowledge guidance knowledge distillation, comprising the steps of inputting images into a lightweight model and a heavy model which are arranged side by side; fusing two image processing results into a new image through a feature pyramid and knowledge guidance knowledge distillation method; in a feature fusion distillation stage, a feature pyramid is formed based on feature map information generated in different stages, and a corresponding feature map result is fused through knowledge distillation; in a knowledge guidance correction stage, a guidance knowledge technology is used to correct the error difference of the knowledge in the feature pyramid fusion process; and in an adaptive adjustment stage, a multi-dimensional loss calculation is performed on the fused feature map, and when the loss is calculated, an adaptive matching mode is used to set the coefficient of the loss function. The application reduces the architecture difference and model capability difference between the lightweight model and the heavy model, increases the robustness of the model, and solves the problem of poor performance of a small sample data set.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image processing method based on feature pyramid and knowledge-guided knowledge distillation. BACKGROUND

[0002] Knowledge distillation is one of the most important model compression techniques in the field of computer vision in current artificial intelligence. Compared with low-rank decomposition, weight quantization, pruning and other model compression methods, knowledge distillation is more simple and universal, and can complete high-quality image synthesis effect under the condition of not changing the model structure. In recent years, the rapid development of artificial intelligence has made great breakthroughs in knowledge distillation technology. At present, the light model through knowledge distillation can approach the accuracy and performance of large models on various general-purpose data sets, and has good performance in various fields (such as classification, target detection, semantic segmentation, etc.). With the advent of the era of artificial intelligence 2.0, computer vision is given more missions, the most critical of which is to enable computer vision to more accurately understand human subjective thoughts and better serve humans. In this environment, large computer vision tasks are currently developing in the direction of continuous optimization, that is, large computer vision models with high computing power can implement corresponding visual tasks and practical applications according to human intentions. However, in edge computing, mobile computing and other aspects, it is impossible to have high computing power and deploy large computer vision models, so the knowledge distillation technology has a good promoting effect on improving the practicality of computer vision technology and promoting the application software of computer vision. In addition, the knowledge distillation technology also makes the computer vision field more intelligent, thereby further promoting the development of artificial intelligence.

[0003] The existing knowledge distillation technology has unsatisfactory actual performance. Most of the knowledge distillation technologies are still through the initial Logits layer knowledge distillation technology, by introducing the "temperature" control factor, and further distilling the intermediate layers of the model, that is, when the cross-entropy function is used to calculate the model loss during the classification task, the "temperature" control factor is added in the SoftMax function to enable the model to retain more knowledge; during the process of processing the entire image by the model, the feature maps at different stages are obtained, and the feature maps at different stages are distilled; the above two are combined into one, and the model is distilled at the same time, so that the lightweight model achieves higher accuracy. In addition, some knowledge distillation technologies introduce the technology of contrast learning, such as defining anchor points, positive samples and negative samples; through the process of knowledge distillation, the similarity of the anchor points and the positive samples is compared, and the difference between the positive samples and the negative samples is compared, different loss functions are obtained, and the two are combined into one to propose a noise contrast estimation loss function. However, in the above method, simply using the feature maps during the process of processing the image by the model will result in poor distillation effect due to the differences in architecture and model capability between the lightweight model and the heavy model. In addition, by using the distillation method of contrast learning, the model can learn the similarity and difference between samples, but due to the lack of sample quantity, the contrast learning method cannot fully learn on a small data set, so that the distillation result is poor. In some specific fields in real life, such as medical treatment, military and industry, it is difficult to obtain a large amount of learning data, so the contrast learning distillation method is no longer applicable. SUMMARY

[0004] To solve the above problems, the present application provides an image processing method based on feature pyramid and knowledge guided knowledge distillation, which reduces the differences in architecture and model capability between the lightweight model and the heavy model, achieves excellent distillation results, increases the robustness of the model, and is suitable for various data sets. It also solves the problem of poor performance on small sample data sets.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is: an image processing method based on feature pyramid and knowledge guided knowledge distillation, characterized in that it comprises the following steps:

[0006] S10, inputting the image into the lightweight model and the heavy model arranged side by side respectively, and processing to obtain two corresponding image processing results;

[0007] S20, combining the two image processing results by the knowledge distillation method based on feature pyramid and knowledge guidance, and fusing the feature maps at different levels together to form a new image; comprising the following steps:

[0008] S201, feature fusion distillation stage, based on the feature map information generated in different stages to form a feature pyramid, and using knowledge distillation to fuse the corresponding feature map results;

[0009] S202, knowledge-guided correction stage, using guided knowledge technology to correct the error difference of the feature pyramid fusion process knowledge;

[0010] S203, adaptive adjustment stage, multi-dimensional loss calculation is performed on the fused feature map, and the coefficient of the loss function is set in an adaptive matching manner when calculating the loss.

[0011] Further, the heavy model adopts a ResNet152 model, and the light model adopts a Resnet50 model.

[0012] Further, in the step S201, the feature fusion distillation stage, based on the feature map information generated in different stages to fuse the corresponding feature map results; the low-level texture information and high-level semantic information output by the light model and the heavy model are fused together, and the originally directly distilled feature maps are fused and distilled using knowledge distillation.

[0013] Further, in the output fusion process of the light model and the heavy model: using mean square error MSE as the loss function, calculating the difference between each pair of feature maps output by the light model and the heavy model after feature pyramid fusion, using SoftMax function to calculate the weight Weight[i] of the corresponding pair, and multiplying the loss value by the corresponding weight to obtain the total loss, called FPLoss;

[0014] Put each pair of feature maps after the feature pyramid into the feature attention module to obtain AT_Weight[i], and use the same method as FPLoss to calculate the weight to obtain the loss after feature attention processing, called SELoss;

[0015] According to FPLoss and SELoss, the feature pyramid distillation loss is obtained; the feature fusion distillation is performed using the feature pyramid distillation loss.

[0016] Further, the calculation method of FPLoss is:

[0017]

[0018] Wherein respectively represent the i-th feature map of all feature maps after the feature pyramid block of the heavy model and the light model, N is the batch size, and the weight Weight[i] represents the i-th loss weight of each pair .

[0019] Further, the calculation method of Weight[i] is:

[0020]

[0021] Further, the calculation method of SELoss is:

[0022]

[0023] wherein respectively represent the i-th feature map of all feature maps after the feature pyramid block of the teacher model and the student model through the feature attention module, N is the batch size, and the weight AT_Weight[i] represents the i-th loss weight of each pair.

[0024] Further, in the step S203, the adaptive loss calculation stage, the ordinary loss, the spatial and channel dimension loss of the fused feature map are calculated, the corresponding weight is calculated by using the SoftMax function, and the total loss is obtained by multiplying the loss value by the corresponding weight. In the process of back propagation, the loss weight is automatically adjusted, and the model learning is optimized.

[0025] The beneficial effects of using the technical solution are:

[0026] The present application uses feature pyramid fusion technology to fuse the feature maps corresponding to each stage of the model processing image, uses knowledge guiding technology to control the errors of the feature pyramid fusion image, and uses the spatial and channel dimension attention mechanism to control the diversity of the fusion image. The present application reduces the architecture difference and model capability difference between the lightweight model and the heavy model through the feature pyramid fusion technology and the knowledge guiding technology, so as to achieve excellent distillation results. Through the spatial and channel dimension attention mechanism, the robustness of the model is increased, so as to be suitable for various data sets. At the same time, the problem of poor performance of knowledge distillation through the contrast learning method in small sample data sets is solved. Therefore, the present application will realize a kind of efficient, simple and universal knowledge distillation technology. The technology can achieve the result of knowledge distillation under the condition of completely meeting various data sets, which has a great role in promoting the development of knowledge distillation technology and promoting computer vision to be more intelligent.

[0027] ​The application fuses the feature pyramid and the knowledge distillation technology, improves the robustness and universality of the feature map in the knowledge distillation process, fuses the feature map after the feature pyramid, the channel attention, the spatial attention and the knowledge distillation, uses the knowledge guiding technology to reduce the error difference generated in the fusion process of the feature pyramid and the knowledge distillation, uses the fusion feature calculation, and is different from the linear addition of the logits layer and the feature map layer in the past, uses the mixed calculation, the mixed calculation channel and the spatial dimension attention Loss loss value, and uses the adaptive loss calculation and the automatic matching mode, and does not manually set the Loss loss corresponding coefficient.

[0028] The application can improve the accuracy of image recognition and image classification, because the knowledge distillation is effectively fused into the whole model processing process by using the feature pyramid technology, and plays a key role, and the knowledge guiding technology reduces the error difference generated in the feature fusion process, so that the image recognition and image classification result can basically meet the expectation. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 It is a flowchart of the image processing method based on the feature pyramid and the knowledge guiding knowledge distillation of the application.

[0030] Figure 2 It is a model principle diagram of the image processing method based on the feature pyramid and the knowledge guiding knowledge distillation in the embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described below with reference to the drawings.

[0032] In this embodiment, referring to Figure 1 The application proposes an image processing method based on the feature pyramid and the knowledge guiding knowledge distillation, which comprises the following steps:

[0033] S10, the image is respectively input into the lightweight model and the heavy model arranged side by side, and two corresponding image processing results are obtained by processing respectively;

[0034] S20, the two image processing results are combined together by the knowledge distillation method based on the feature pyramid and the knowledge guiding knowledge distillation, and different levels of feature maps are fused into a new image; comprising the following steps:

[0035] S201, feature fusion distillation stage, the feature pyramid is formed based on the feature map information generated in different stages, and the corresponding feature map result is fused by using the knowledge distillation;

[0036] S202, knowledge guiding correction stage, using guiding knowledge technology to correct the error difference of feature pyramid fusion process knowledge;

[0037] S203, adaptive adjustment stage, multi-dimensional loss calculation is performed on the fused feature map, and when calculating the loss, the coefficients of the loss function are set in an adaptive matching manner.

[0038] As an optimization scheme of the above embodiment, the heavy model adopts a ResNet152 model, and the light model adopts a Resnet50 model.

[0039] As an optimization scheme of the above embodiment, in the step S201, the feature fusion distillation stage, based on the feature map information generated in different stages, the corresponding feature map results are fused; the low layer texture information and high layer semantic information output by the light model and the heavy model are fused together, and knowledge distillation is combined to perform feature fusion distillation on the feature map which is originally directly distilled.

[0040] As shown in Figure 2 , in the output fusion process of the light model and the heavy model: using mean square error MSE as the loss function, the difference between each pair of feature maps output by the light model and the heavy model after feature pyramid fusion is calculated, the loss function values of all pairs are calculated using SoftMax function to calculate the weight Weight[i] of the corresponding pair, and the loss value is multiplied by the corresponding weight to obtain the total loss, called FPLoss.

[0041] The calculation method of FPLoss is:

[0042]

[0043] Wherein respectively represent the i-th feature map of all feature maps after the feature pyramid block of the heavy model and the light model, N is the batch size, and the weight Weight[i] represents the i-th loss weight of each pair .

[0044] The calculation method of Weight[i] is:

[0045]

[0046] Put each pair of feature maps after the feature pyramid into the feature attention module to obtain AT_Weight[i], and use the same method as FPLoss to calculate the weight to obtain the loss after feature attention processing, called SELoss.

[0047] The calculation method of SELoss is:

[0048]

[0049] wherein The i-th feature map of all feature maps after the feature pyramid block of the teacher model and the student model respectively represents the feature map after the feature attention module, N is the batch size, and the weight AT_Weight[i] represents the i-th loss weight of each pair .

[0050] According to the FPLoss and the SELoss, the FPLoss and the SELoss can be added by using a weight coefficient, so as to obtain a feature pyramid distillation loss; and the feature fusion distillation is performed by using the feature pyramid distillation loss.

[0051] As an optimization scheme of the above embodiment, in the step S203, an adaptive loss calculation stage, the normal loss, the spatial and channel dimension loss of the fused feature map are calculated, the weight of the corresponding pair is calculated by using a SoftMax function, and the loss value is multiplied by the corresponding weight to obtain a total loss, in the process of back propagation, the loss weight is automatically adjusted, and the model learning is optimized.

[0052] The basic principles and main features of the present application and the advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and the description in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An image processing method based on feature pyramids and knowledge-guided knowledge distillation, characterized in that, Including the following steps: S10, Input the images into the lightweight model and the heavy model that are side by side, and process them to obtain two corresponding image processing results; S20, the two image processing results are combined into a new image by using a knowledge distillation method based on feature pyramid and knowledge guidance, which combines feature maps at different levels. include step: S201, Feature Fusion Distillation Stage: Based on the feature map information generated in different stages, a feature pyramid is constructed, and knowledge distillation is used to fuse the corresponding feature map results. S202, Knowledge-guided correction stage, uses guided knowledge technology to correct erroneous discrepancies in knowledge during the feature pyramid fusion process; S203, Adaptive Adjustment Stage: The fused feature map is subjected to multi-dimensional loss calculation, and the coefficients of the loss function are set using adaptive matching when calculating the loss. In the process of fusion of outputs from lightweight and heavy models: the mean squared error (MSE) is used as the loss function to calculate the difference between the feature maps of each pair of lightweight and heavy model outputs after feature pyramid fusion. The loss function values ​​of all pairs are used to calculate the weights of the corresponding pairs using the SoftMax function, and the loss values ​​are multiplied by the corresponding weights to obtain the total loss, which is called FPLows. Each pair of feature maps after the feature pyramid is put into the feature attention module to obtain AT_Weight[i], and the weights are calculated using the same method as FPLoss to obtain the loss after feature attention processing, which is called SELows; Based on FPLoss and SELoss, the characteristic pyramid distillation loss is obtained; Feature fusion distillation is performed using feature pyramid distillation loss.

2. The image processing method based on feature pyramids and knowledge-guided knowledge distillation according to claim 1, characterized in that, The heavy model uses the ResNet152 model, and the lightweight model uses the ResNet50 model.

3. The image processing method based on feature pyramids and knowledge-guided knowledge distillation according to claim 1, characterized in that, In step S201, during the feature fusion distillation stage, the corresponding feature map results are fused based on the feature map information generated at different stages; the low-level texture information and high-level semantic information output by the lightweight model and the heavy model are fused together, and at the same time, combined with knowledge distillation, the feature maps that were originally distilled directly are subjected to feature fusion distillation.

4. The image processing method based on feature pyramids and knowledge-guided knowledge distillation according to claim 1, characterized in that, The method for calculating FPLloss: in These represent the i-th feature map of all feature maps after the feature pyramid block of the heavy model and the lightweight model, respectively. N is the batch size, and Weight[i] represents the weight of each pair. The i-th loss weight.

5. The image processing method based on feature pyramids and knowledge-guided knowledge distillation according to claim 4, characterized in that, The calculation method for Weight[i] is as follows:

6. The image processing method based on feature pyramid and knowledge-guided knowledge distillation according to any one of claims 3-5, characterized in that, The method for calculating SELows: in These represent the i-th feature map after passing through the feature attention module, representing all feature maps following the feature pyramid blocks of the teacher and student models, respectively. N is the batch size, and AT_Weight[i] represents the weight of each pair. The i-th loss weight.

7. The image processing method based on feature pyramids and knowledge-guided knowledge distillation according to claim 1, characterized in that, In step S203, during the adaptive loss calculation stage, the ordinary loss, spatial and channel dimension loss of the fused feature map are calculated. The SoftMax function is used to calculate the weights of the corresponding pairs, and the loss value is multiplied by the corresponding weights to obtain the total loss. During the backpropagation process, the loss weights are automatically adjusted to optimize model learning.

Images