Cross-weighted distillation method based on morphological information guidance

Abstract

The invention relates to a cross-weighted distillation method based on morphological information guidance, and the method comprises the steps: carrying out the guidance of the high-level features of a student model through the low-level features of a teacher model, and carrying out the non-adjacent-level features; the current hierarchy of the students is supervised by utilizing the bottom layer and current hierarchy features in the teacher model, and weighted sum of supervised constraints is obtained by adopting a non-fusion mode; and meanwhile, a cross attention mechanism is adopted to weight the multi-level guidance constraint. According to the method, guidance of low-level morphological features of the teacher network on high-level semantic features of the student network is fully utilized, the overall performance of the student network can be effectively improved, meanwhile, the added cross attention mechanism enables the student network to learn key features in a targeted mode, interference on redundant information is reduced, and distillation performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target visual detection, and in particular to a cross-weighted distillation method guided by morphological information. Background Art

[0002] Based on the characteristic analysis of samples in industrial scenarios, it is found that the semantic features of sample defects are usually not significant enough, while manual subjective judgment often relies on the morphological information of these defects, which highlights the importance of low-level features in the recognition task. Therefore, the design of knowledge distillation needs to focus on the feature transfer channel between the teacher model and the student model to ensure that not only high-level semantic features can be efficiently transferred during the distillation process, but also low-level detailed features can be accurately captured and transferred.

[0003] For feature-based knowledge distillation, in addition to the one-to-one distillation at the same level, some studies have explored the distillation effect across levels and verified it in classification tasks, and proposed a knowledge distillation scheme ReviewKD based on the knowledge review strategy. By gradually fusing the low-level features of the teacher upwards to guide the distillation of the student model. Although Patent 202111174278.7 proposes to perform distillation on multi-level features separately, representing the low-level, middle-level, and high-level feature information of the model at different levels respectively. However, these methods still have some limitations for the distillation of defect detection tasks in industrial scenarios. First, for the ReviewKD scheme, the object of distillation is the fusion of adjacent low-level information and current-level information for distillation, and the morphological information in the low-level features is not essentially utilized because it is impossible to determine whether it has been replaced by high-level semantic information during the fusion process;

[0004] The scheme proposed in Patent 202411654869.8 is still the feature information distillation between the same levels of the teacher and student models, and does not use the low-level information to enhance and supplement the high-level features of the model itself across levels. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to provide a cross-weighted distillation method guided by morphological information, which uses the low-level features of the teacher network to guide the distillation of the high-level features of the student network, so as to realize the effective fusion of semantic information and morphological information.

[0006] The technical solution adopted by the present invention to solve its technical problems is: a cross-weighted distillation method guided by morphological information, including the following steps,

[0007] S1. Input a set of industrial defect detection data sets, and after preprocessing, input them into the teacher network and the student network respectively;

[0008] S2. Select high-level features fT from the multi-level features of the teacher network nand the low-level feature fT n-k , where k is a fixed layer spacing;

[0009] S3. Adjust the channel dimension through a convolution operation to make the target feature layer fS of the student network n match the number of channels of fT n . The alignment of the channels of the student feature to the channels of the teacher feature is achieved by a convolution module; adjust the spatial resolution through bilinear interpolation to make fS n match the size of fT n ;

[0010] S4. Calculate the distillation loss of the teacher's high-level feature on the student feature:

[0011] S5. Calculate the distillation loss of the teacher's low-level feature on the student feature:

[0012] S6. In the distillation process, the student model will set the alignment of multiple layers of features to the teacher model, combine multi-level information, and calculate the final cross-layer guidance loss in an accumulative manner:

[0013] S7. Calculate the attention-weighted distribution to obtain the loss of the final feature distillation part.

[0014] Furthermore, in step S3 of the present invention, the alignment method of the channels of the student feature to the channels of the teacher feature is:

[0015]

[0016] where the shape of the student fS n is b*c*w*h, and the shape of the teacher fT n is b*C*W*H;

[0017] The method of interpolating the student's feature is:

[0018]

[0019] where,

[0020] β x =x - j, β y =y - i

[0021] i = int(y), j = int(x)

[0022] x = -0.5+(u + 0.5) / s W , y = -0.5+(v + 0.5) / s H

[0023] s H =H / h, s W =W / w

[0024] Let \(w\) and \(h\) be the width and height of the student features respectively, and \(W\) and \(H\) be the width and height of the teacher features respectively.

[0025] Furthermore, in step S8 of the present invention, the following steps are included:

[0026] 1) Extract the feature map \(f_T\) of the \(n\)-th layer of the teacher network n and the feature map \(f_S\) of the \(n\)-th layer of the student network n ;

[0027] 2) Calculate the autocorrelation matrix \(A_T\) for the teacher network features n , and the teacher feature \(f_T\) n is transformed from the feature map \(b*c*h*w\) to \(b*chw\) via the matrix transformation operation \(R(.)\):

[0028] 3) Calculate the autocorrelation matrix \(A_S\) for the student network features n , and the \(n\)-th layer feature map \(f_S\) of the student n first undergoes the diagnostic transformation \(R(.)\):

[0029] 4) Use a multi-layer perceptron (MLP) to denoise \(A_T\) n and \(A_S\) n ;

[0030] 5) Calculate the weight coefficient \(w\) between the feature map matrices of the teacher network and the student network (sn,tn) ;

[0031] 6) Multiply all the weight features involved in the calculation in the teacher network by their weights to obtain a comprehensive feature, and use this feature to calculate the MSE loss with the corresponding feature of the student network.

[0032] Furthermore, in step 2) of the present invention, the way that the matrix transformation operation \(R(.)\) transforms the feature map from \(b*c*h*w\) to \(b*chw\) is:

[0033] \(R(f_T\) n ) = reshape(f_T\) n ) ∈ R b*chw

[0034] Then obtain the similarity matrix of the teacher feature map itself:

[0035] \(A_T\) n = R(f_T\) n ).(R(f_T\) n )) T ∈ R b*b

[0036] where \(A_T\) nIt represents the similarity information among each sample in the current input batch data with respect to the features of the teacher at the n-th layer.

[0037] Furthermore, in step 3) of the present invention, for the feature map fS of the student at the n-th layer n First, it is processed by the diagnostic transformation R(.) as follows:

[0038] R(fS n ) = reshape(fS n ) ∈ R b*chw

[0039] Then, similarly, the similarity matrix of the student features is calculated to obtain the mutual similarity relationship of each sample at the n-th layer of the student model:

[0040] AS n = R(fT n ).(R(fS n )) T ∈ R b*b .

[0041] Furthermore, in step 4) of the present invention, the denoising process specifically performs a single feed-forward calculation (FFN) using an MLP:

[0042] QS n = MLP(AS n )

[0043] KT n = MLP(AT n ).

[0044] Furthermore, in step 5) of the present invention, the weight coefficient is calculated by taking the dot product of QS n and KT n and is normalized using the Softmax function:

[0045]

[0046] The weight coefficient w (sn,tn) represents the correlation strength between the feature map of the n-th layer of the student network and the feature map of the n-th layer of the teacher network.

[0047] The beneficial effect of the present invention is to solve the defects existing in the background technology.

[0048] In the object detection task, a method based on distillation of multi-level low-level features of the teacher network and a single high-level feature of the student network is proposed; a cross-attention mechanism is introduced to assign adaptive weights to features at different levels to help the student network determine the importance of which features of the teacher network are guiding for the current layer.

[0049] Adopt a non-fusion method to utilize multi-level feature information for supervision and distillation, fully utilize the main morphological information in the low layer to supplement the high-level semantic information, and at the same time use cross-attention weighting to adaptively evaluate the guiding weights of multiple levels, reducing the redundant information brought by multi-level information.

[0050] Fully utilize the guidance of the low-layer morphological features of the teacher network to the high-layer semantic features of the student network, which can effectively improve the overall performance of the student network. At the same time, the added cross-attention mechanism enables the student network to learn key features specifically, reduces the interference of redundant information, and improves the distillation performance. Brief Description of the Drawings

[0051] Figure 1 is the overall method flow chart of the present invention;

[0052] Figure 2 is the cross-layer guidance learning flow chart of the present invention;

[0053] Figure 3 is the cross-attention weight calculation flow chart of the present invention;

[0054] Figure 4 is the overall framework flow schematic diagram of the implementation of the method of the present invention;

[0055] Figure 5 is the distillation architecture schematic diagram of the teacher model guiding the student model across layers of the present invention;

[0056] Figure 6 is the visualization result of defect detection on metal material data of the present invention;

[0057] Figure 7 is the visualization result of defect detection on the image data of the front cover of a mobile phone of the present invention;

[0058] Figure 8 is the visualization result of defect detection on the dark field image data of the glass cover plate of the present invention. Detailed Embodiment

[0059] Now, the present invention will be further described in detail with reference to the accompanying drawings and preferred embodiments. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0060] In industrial defect detection tasks, the semantic features of defect samples are often weakened, while morphological features play an important role in recognition and classification. However, due to its simplified structure, the student network may have difficulty retaining sufficient morphological information in high-level features. In contrast, the teacher network has a more complex structure and can capture richer morphological features at the low level. Therefore, making full use of the guidance of the low-level morphological features of the teacher network on the high-level semantic features of the student network can effectively improve the overall performance of the student network. The core idea of the cross-layer guidance strategy is to jointly guide the feature learning of the nth layer of the student network by combining the high-level features of the nth layer and the low-level features of the nth - k layer (k is a constant) of the teacher network, so as to retain key morphological features while strengthening the semantic features of the student network. This strategy is different from traditional one-to-one hierarchical distillation, emphasizing the complementarity of cross-layer features and enhancing the distillation effect.

[0061] Such as Figures 1-8 The cross-weighted distillation method based on morphological information guidance shown, includes the following steps,

[0062] Input a set of industrial defect detection datasets, and after preprocessing, input them into the teacher network and the student network respectively.

[0063] Select high-level features fT n and low-level features fT n-k from the multi-level features of the teacher network, where k is a fixed layer spacing. The list of high-level features can be [15, 18, 21]. The high-level features are selected as the feature layers for the teacher to conduct knowledge distillation to the student. The corresponding list of low-level features for the high-level feature list is [4, 6, 8], that is, the 15th layer of the student model will be guided by the 15th layer and the 4th layer of the teacher model, the 18th layer of the student model will be guided by the 18th layer and the 6th layer of the teacher model, and the 21st layer of the student model will be guided by the 21st and 8th layers of the teacher model;

[0064] In the feature map, b, c, h, and w represent the batch dimension, channel dimension, height, and width respectively.

[0065] Since the target feature layer fS n of the student network may have different numbers of channels and spatial resolutions from fT n and fT n-k of the teacher network, feature alignment is performed:

[0066] Adopt convolution operation to adjust the channel dimension to make fS n match the number of channels of fT n . Specifically: Given that the shape of student fS n is b*c*w*h, and the shape of teacher fT nThe shape is b*C*W*H,

[0067] First,

[0068] The alignment of the channels of the student features to the teacher feature channels is achieved by the convolution module:

[0069]

[0070] In addition to the difference in the number of channels of the feature dimensions, the width and height of the feature maps of different layers will also be different. Therefore, alignment is also required in these two directions. This method uses bilinear interpolation to adjust the spatial resolution so that fS n matches fT n in size.

[0071] Specifically: Align the width and height w and h of the student features to the width and height W and H of the teacher features. First, determine the scaling ratio:

[0072] s H = H / h, s W = W / w

[0073] For all coordinate positions u and v in the w and h directions, the transformed coordinates x and y are:

[0074] x = -0.5 + (u + 0.5) / s W , y = -0.5 + (v + 0.5) / s H

[0075] Convert them to the nearest integers:

[0076] i = int(y), j = int(x)

[0077] The interpolation weights based on the nearest neighbor distances are:

[0078] β x = x - j, β y = y - i

[0079] Based on the above formulas, the features of the student can be interpolated, that is:

[0080]

[0081] After the alignment of the size of the feature map and the dimension of the feature channels, the features of the student can maintain the same size as the teacher features to be guided. All the student features fS n in the following article are defaulted to be after feature alignment. Therefore, the supervision error for guiding by the high-level features of the teacher can be expressed as:

[0082]

[0083] The supervision error for guiding the low-level features of the teacher is:

[0084]

[0085] The comprehensive cross-layer guidance constraint for a single layer of students is:

[0086]

[0087] During the distillation process, the student model will set the features of multiple layers to align with the teacher model. Therefore, the distillation loss under the current strategy is the accumulation of multiple cross-layer guidance losses, that is:

[0088]

[0089] The list of features inside [n1, n2, …, n n Specifically, it is [15, 18, 21].

[0090] The cross-layer guidance strategy proposed in this method is different from the previous one-to-one guidance distillation and the multi-layer knowledge review distillation method mentioned above. Compared with fusing the teacher layer information to be distilled before distilling a specific layer of the student, this method chooses not to fuse, but to accumulate all supervision constraints, making the most of the semantic information in the high-level features and the morphological-related information in the low-level features in the teacher model. Compared with using a fusion module, it more directly proves that the low-level morphological information still has guiding value for the high-level semantic information, and uses the complementarity of such multi-level features to improve the effectiveness of the entire distillation. Of course, the cost of doing this is to bring a large number of loss terms. The direct accumulation of different loss terms will increase the convergence difficulty of the student during the learning process. In addition, to maintain the performance of feature-based knowledge distillation, it is usually hoped that there is semantic or feature correlation in the feature transfer between the student network and the corresponding teacher network. If there is no correlation between the guidance layer of the teacher network and the layer to be guided of the student network, then it will inevitably affect the distillation effect of the model or bring negative impacts. Next, this method adds a strategy of adaptively weighting the loss terms of multiple-level features on the basis of cross-layer guidance, and balances the loss terms of multiple guidance layers through weight allocation.

[0091] In the feature-based knowledge distillation task, the teacher network constrains the student network by implementing on the intermediate feature layer. To reduce the influence of multiple guidance layers of the teacher on a certain layer of the student, when designing the guidance strategy, this method adds a calculation method based on attention weights to obtain the guidance weights of the teacher's multi-level features on the single-level features of the student network.

[0092] Inspired by the self-attention in the Transformer, which can calculate the correlation between different vectors. By virtue of this idea, during the distillation process, each layer of the teacher network and the student network is regarded as a vector representation, and the dot product of these representation vectors is calculated to obtain the weight distribution between the teacher network and the student network. Finally, these weight distributions are applied to the corresponding feature layers as the weights for each layer, that is, the selection of feature layers is performed.

[0093] Specifically, in the process of calculating the feature correlation, first calculate the similarity matrix A of the feature map. First, fT n represents the feature map of the nth layer of the teacher network, and fS n represents the feature map of the nth layer of the student network. The teacher feature fT n is transformed from the feature map of b*c*h*w to b*chw through the matrix transformation operation R(.):

[0094] R(fT n ) = reshape(fT n ) ∈ R b*chw

[0095] Then obtain the similarity matrix of the teacher feature map itself:

[0096] AT n = R(fT n ).(R(fT n )) T ∈ R b*b

[0097] Among them, AT n represents the similarity information between each sample in the currently input batch data in the teacher's nth layer feature.

[0098] Similarly, for the nth layer feature map fS of the student n first perform the diagnostic transformation R(.) processing as:

[0099] R(fS n ) = reshape(fS n ) ∈ R b*chw

[0100] Then similarly calculate the similarity matrix of the student features to obtain the mutual similarity relationship of each sample in the nth layer of the student model:

[0101] AS h = R(fT n ).(R(fS n )) T ∈ R b*b

[0102] Similarly referring to the Transformer architecture, a multi-layer perceptron (MLP) module is introduced to process the similarity matrices of the student and the teacher, removing the noise in the similarity matrices. Specifically, a feed-forward calculation (FFN) is performed using the MLP:

[0103] QS n = MLP(AS n )

[0104] KT n = MLP(AT n )

[0105] Next, the weight coefficient w between the feature map matrices of the teacher network and the student network is calculated. (sn,tn) , and this weight coefficient is calculated by taking the dot product of QS n and KT n , and is normalized using the Softmax function:

[0106]

[0107] This weight coefficient w (sn,tn) represents the correlation strength between the feature map of the n-th layer of the student network and the feature map of the n-th layer of the teacher network. This weight assignment mechanism based on the attention mechanism can, to a certain extent, suppress the negative impact of the shape mismatch between the teacher feature layer and the student feature layer during the distillation process, and can apply a greater weight to the guidance of highly correlated features.

[0108] Finally, all the weight features involved in the calculation in the teacher network are multiplied by their weights to obtain a comprehensive feature, and the MSE loss is calculated using this feature and the corresponding feature of the student network.

[0109] L CrossLayer = w (sn,tn,tn-k) * (‖fT n - fS n ‖2 + ‖fT n-k - fS n ‖2)

[0110] Combined with the previous formula, the loss of the final feature distillation part is:

[0111]

[0112] For the detection task, the original task-specific loss L Task follows the default configuration of the model:

[0113] L Task = L cls + L box + L obj #

[0114] The total loss is:

[0115] L Total = L MCWD + L Task #(3 - 9)

[0116] Example 1

[0117] As Figure 6 shown, the visualization of the defect detection results on the metal material data. From left to right, they are the true annotation (ground truth), the teacher model, the student model, and the visualization result display of the patented solution. On this data, the teacher model had inaccurate predictions, obvious false positive detections (lines 1 - 4) and one false negative instance (line 5). The student model was more prone to false negative results, i.e., missing detected defects (lines 1, 2, and 5). After optimization by the invention solution, through the reinforcement learning of the morphological information provided by the teacher model, the performance of the distilled model was improved.

[0118] Example 2

[0119] As Figure 7 shown, the visualization of the defect detection results on the front cover image data of the mobile phone. From left to right, they are the true annotation (ground truth), the teacher model, the student model, and the visualization result display of the patented solution. On this data, the teacher model had false positive detections (lines 1, 2) and false negative results (lines 3, 4). Similarly, the student model still had more missed detection samples (lines 2, 3, 4). Similarly, the accuracy of the student model was improved after distillation.

[0120] Example 3

[0121] As Figure 8 shown, the visualization of the defect detection results on the dark field image data of the glass cover plate. From left to right, they are the true annotation (ground truth), the teacher model, the student model, and the visualization result display of the patented solution. On this data, the teacher and student models respectively had cases of misidentifying the defect categories, while the distilled model after enhancing the low - level morphological information made predictions consistent with the annotation, reflecting the importance of defect morphological information for category recognition in industrial scenarios.

[0122] The results of the effect comparison experiment are shown in the following table:

[0123]

[0124]

[0125] As can be seen from the above table, the proposed method shows excellent performance on all datasets compared with other methods, which verifies its effectiveness in improving detection accuracy while maintaining computational efficiency.

[0126] To evaluate the contributions of the cross-layer guidance strategy (Cross-layer Learning) and cross-attention weighting (Attention Weighted) used in the study, an ablation study was conducted. The experimental results of module ablation are shown in the following table:

[0127]

[0128] The results highlight the roles of the two strategies in improving performance. When only the cross-layer guidance strategy is enabled, the student model shows a certain improvement compared with when no strategy is used. This performance improvement verifies the effectiveness of the cross-layer guidance strategy in enhancing the expression of high-level semantic features through low-level morphological information. Especially in the industrial defect detection task with significant sample morphological features, the complementarity of cross-layer features has obvious advantages. When the two strategies are applied jointly, the performance of the student model improves most significantly compared with the basic student model. This indicates that the cross-attention weighting strategy can more accurately allocate the importance weights of teacher features in combination with the cross-layer guidance strategy, enabling the student network to more effectively learn the key features of the teacher network.

[0129] What is described in the above specification is only the specific implementation manners of the present invention. Various illustrative examples do not constitute limitations on the essence of the present invention. Those of ordinary skill in the art can make modifications or variations to the previously described specific implementation manners after reading the specification without departing from the essence and scope of the invention.

Claims

1. A cross-weighted distillation method guided by morphological information, characterized in that: including the following steps, S1. Input a set of industrial defect detection datasets, and after preprocessing, input them into the teacher network and the student network respectively; S2. Select the high-level feature fT and the low-level feature fT from the multi-level features of the teacher network, where k is a fixed layer spacing; n n-k ​​ S3. Adjust the channel dimension using a convolution operation to make the target feature layer fS of the student network n match fT n in terms of the number of channels. The alignment of the channels of the student features to the teacher feature channels is achieved by a convolution module; adjust the spatial resolution using bilinear interpolation to make fS n match fT n in size; S4. Calculate the distillation loss of the teacher's high-level features to the student's features: S5. Calculate the distillation loss of the teacher's low-level features to the student's features: S6. During the distillation process, the student model will set multiple layers of features to align with the teacher model. Combining multi-level information, calculate the final cross-layer guidance loss in an accumulative manner: S7. Calculate the attention-weighted distribution to obtain the loss of the final feature distillation part.

2. The cross-weighted distillation method guided by morphological information according to claim 1, characterized in that: In the step S3, the alignment method of the student's feature channels to the teacher's feature channels is: Among them, the student fS n has a shape of b*c*w*h, and the teacher fT n has a shape of b*C*W*H; The method of interpolating the student's features is: Wherein, β x = x - j, β y = y - i i = int(y), j = int(x) x = -0.5 + (u + 0.5) / s W , y = -0.5 + (v + 0.5) / s H s H = H / h, s W = W / w w and h are the width and height of the student's features respectively, and W and H are the width and height of the teacher's features respectively.

3. The cross-weighted distillation method guided by morphological information as claimed in claim 1, wherein: In the step S8, it includes the following steps, 1) Extract the feature map fT of the n-th layer of the teacher network n and the feature map fS of the n-th layer of the student network n ; 2) Calculate the autocorrelation matrix AT for the teacher network features n , the teacher feature fT n Convert the feature map from b*c*h*w to b*chw through the matrix transformation operation R(.): 3) Calculate the autocorrelation matrix AS for the student network features n For the n-th layer feature map fS of the student n First, perform the diagnostic transformation R(.): 4) Use a multi-layer perceptron (MLP) to denoise AT n and AS n ; 5) Calculate the weight coefficient w between the feature map matrices of the teacher network and the student network (sn,tn) ; 6). Multiply all the weight features participating in the calculation in the teacher network by their weights to obtain a comprehensive feature, and use this feature to calculate the MSE loss with the corresponding student network's feature.

4. The cross-weighted distillation method guided by morphological information according to claim 3, wherein: In the step 2), the matrix transformation operation R(.) converts the feature map from b*c*h*w to b*chw in the following way: R(fT n ) = reshape(fT n ) ∈ R b*chw Then obtain the similarity matrix of the teacher's feature map itself: AT n = R(fT n ).(R(fT n )) T ∈R b*b Among them, AT n represents the similarity information among each sample in the current input batch data with respect to the features of the teacher at the n-th layer.

5. The cross-weighted distillation method guided by morphological information according to claim 3, wherein: In the said step 3), the n-th layer feature map fS of the student n is first subjected to a diagnostic transformation R(.) and processed as follows: R(fS n ) = reshape(fS n ) ∈ R b*chw Then similarly calculate the similarity matrix of the student's features to obtain the mutual similarity relationship of each sample in the nth layer of the student model: AS n = R(fT n ).(R(fS n )) T ∈R b*b 。 6. The cross-weighted distillation method guided by morphological information according to claim 3, wherein: In the step 4), the denoising process specifically uses MLP for a single feed-forward calculation (FFN): QS n = MLP(AS n ) KT n = MLP(AT n )。 7. The cross-weighted distillation method guided by morphological information according to claim 3, wherein: In the said step 5), the weight coefficient is calculated by computing the dot product of QS n and KT n and is normalized using the Softmax function: Weight coefficient w (sn,tn) Represents the correlation strength between the feature map of the n-th layer of the student network and the feature map of the n-th layer of the teacher network.

Images