Strip steel surface defect detection method based on lightweight double enhanced network

Abstract

The invention discloses a strip steel surface defect detection method based on a lightweight double enhanced network, and belongs to the field of target detection. The invention relates to a novel strip steel surface defect detection framework, which is characterized in that a lightweight module and a feature fusion module are stacked in the framework, mixed knowledge distillation is utilized outside the framework, and a strip steel surface defect detection network which is light enough is constructed. According to a conventional strip steel surface defect detection network ( / teacher model), C2f modules formed by common convolution are stacked to serve as Backbone, and a multi-layer coordination adaptive feature fusion module and a decoupling head are introduced to jointly form a Neck part. The lightweight dual enhancement network ( / student model) is formed by stacking FasterNet modules formed by lightweight convolution as a Backbone, and other structures of the lightweight dual enhancement network are the same as those of a teacher model. According to the internal and external dual enhancement mode, the parameter quantity of the network is greatly reduced, the detection speed is improved, the problem of low precision of a lightweight surface defect detection network is solved, and a better balance can be obtained between the detection precision and the speed.

Description

Technical Field

[0001] The invention relates to the field of target detection, and in particular to a strip steel surface defect detection method based on a lightweight double-enhanced network. Background Art

[0002] Strip surface defect detection is a key link in the production of high-quality steel and a key indicator for product maintenance. Traditional strip surface defect detection technologies are being replaced by machine vision-based detection methods, which have higher accuracy and lower computational costs. With the rapid development of deep learning, the detection accuracy of neural network-based defect detection models has been pushed to new heights. However, existing defect detection algorithms are usually based on large frameworks such as VGG, ResNet, and Transformer. The huge model parameters enable these conventional networks to achieve high detection accuracy, but face challenges in balancing inference speed and model size, which limits their application scenarios. For example, the FasterR-CNN based on Resnet50 has an FPS of only 19 when tested on GTX4070, which is difficult to meet the real-time detection needs of the industry.

[0003] The emergence of lightweight networks has greatly alleviated the limitation of hardware resources. Many researchers are committed to designing lightweight modules. Technologies such as deep separable convolution and channel shuffling have been proposed in order to fundamentally solve the problem of model efficiency. In addition, knowledge distillation, as an effective lightweight method, has been introduced into dense detection tasks to improve the learning ability of lightweight student networks. By using a pre-trained large teacher model to guide the training process of the student model, a lightweight student model with good performance can be obtained. Compared with traditional networks, lightweight networks obtained by lightweight methods have unique advantages in computing resource consumption, operating efficiency and flexibility. However, due to the small number of overall network parameters, the following problems will be brought to lightweight models: 1) How can limited feature information be used efficiently? 2) The small number of parameters limits the overall performance of the model. How to break through the information quantity barrier to improve the performance bottleneck? Summary of the invention

[0004] The purpose of the present invention is to provide a strip surface defect detection method based on a lightweight dual-reinforcement network, which solves the problems of large model size and slow detection speed of conventional surface defect detection networks, improves the problem of low accuracy of lightweight surface defect detection networks, and can achieve a better balance between detection accuracy and speed, providing a more applicable solution for embedded devices and scenarios with limited computing power.

[0005] To achieve the above object, the present invention provides a strip surface defect detection method based on a lightweight double-reinforced network, comprising the following steps:

[0006] S1. Read image samples for strip surface defect detection, process them using data enhancement technology, and then include them in the sample library;

[0007] S2, taking the defect image samples as inputs of the conventional strip surface defect detection network and the lightweight dual enhancement network, respectively, and extracting the feature map of the training image through a feature extraction module stacked by convolutional layers;

[0008] S3, inputting feature mapping information from different levels into a multi-layer coordinated adaptive feature fusion module to ensure efficient fusion and utilization of feature information;

[0009] S4, using a decoupling head to identify and locate feature information;

[0010] S5. The distillation loss function is used to force the lightweight dual reinforcement network to learn the intermediate prompt layer features and the logits output of the classifier of the conventional strip surface defect detection network.

[0011] Preferably, in step S1, the NEU-DET public dataset is used as the dataset, which contains 1,800 images, representing six types of defects, each defect type has 300 images, the size of each image is 200×200 pixels, and the dataset is divided into a training set and a test set in a ratio of 7:3.

[0012] Preferably, in step S2, the conventional strip surface defect detection network uses a C2f module stack composed of ordinary convolution as the Backbone, and introduces a multi-layer coordinated adaptive feature fusion module and a decoupling head to form a Neck part.

[0013] Preferably, in step S2, the lightweight dual enhancement network uses a FasterNet module stack with PConv as the core to form the backbone of the network, introduces a multi-layer coordinated adaptive feature fusion module and a decoupling head to form the Neck part, and finally uses a hybrid knowledge distillation strategy to extract more effective knowledge from the conventional strip surface defect detection network.

[0014] Preferably, in step S3, the multi-layer coordinated adaptive feature fusion module adopts a top-down structure to integrate high-level semantic feature maps, processes feature maps from different levels through the adaptive fusion ASFF module to adjust the weights, and then adopts a bottom-up path to completely fuse the positioning information from the lower layer with the semantic information from the higher layer.

[0015] Preferably, in the adaptive fusion ASFF module, let the feature of layer n be x n , n∈{2,3,4,5}, its adaptive feature fusion is expressed as

[0016]

[0017] in Represents the feature vector at the position (i, j) of the feature map adjusted from different layers to the n layer, Represents the feature vector output along the corresponding channel;

[0018] They represent the adaptive weights of different levels, which are calculated using the Softmax function.

[0019] Preferably, in step S5, the hybrid distillation method adopts a dual-channel strategy, that is, a distillation loss composition based on a feature channel and a logit output channel, as shown below:

[0020] S51, the teacher network and the student network are represented as T and S respectively, and the corresponding activation maps are represented as y T and S , then the distillation loss based on the feature channel can be expressed as follows:

[0021]

[0022] Among them, L(·) represents the cross entropy loss, which is used to evaluate the difference in channel distribution between the teacher network and the student network, and φ(·) represents the conversion of activation values ​​into probability distribution using the Softmax function, as shown below:

[0023]

[0024] Where c = 1, 2, ···, C represents the channel index, i represents the spatial position index of the channel, and T is the temperature parameter, which is expressed as follows using KL divergence:

[0025]

[0026] Where x represents a sample. From the above formula, we can see that when the probability knowledge of the student model is similar to that of the teacher model, the KL divergence value can be minimized.

[0027] S52. The distillation loss based on logit output is divided into two parts: classification distillation loss and positioning distillation loss. The classification distillation loss part uses the binary cross entropy loss function in conjunction with the KL divergence, as shown below:

[0028]

[0029] Where, L BCE represents the binary cross entropy loss, represents the total classification distillation loss, ω = | y T '-y S '| Represents the loss weighting coefficient, which is used to measure the importance of sample x;

[0030] The EIoU loss function is used to locate the distillation loss. EIoU consists of three parts: IoU loss, distance loss, and width and height loss, as shown below:

[0031]

[0032] Among them, w c and h c is the width and height of the minimum closed area covering the predicted bounding box and the true bounding box, and ρ is the Euclidean distance between the two points;

[0033] Like the classification loss, the positioning loss function formula after introducing the weight coefficient is:

[0034]

[0035] In the formula, represents the total positioning distillation loss; therefore, the total distillation loss function based on the logit method is expressed as

[0036]

[0037] Where α and β are hyperparameters used to adjust the loss weights of classification loss and positioning loss respectively;

[0038] Therefore, the total loss function of hybrid distillation is:

[0039]

[0040] Among them, the feature-based distillation loss is expressed as It uses Softmax to normalize the feature map into a probability map, and then minimizes the KL divergence; the logit-based distillation loss is obtained by the binary classification cross entropy loss and EIOU positioning loss The classification loss is calculated by converting the logit map into multiple binary classification maps. The localization loss is calculated using the EIOU value between the predicted bounding boxes of the two models.

[0041] Therefore, the present invention adopts a strip surface defect detection method based on a lightweight double-reinforced network with the above structure, which has the following beneficial effects:

[0042] (1) The present invention uses a FasterNet module stack with PConv as the core to form the backbone of the network, which greatly reduces the number of network parameters and improves the detection speed, solving the problems of large model size and slow detection speed of conventional surface defect detection networks.

[0043] (2) The present invention introduces a multi-layer coordinated adaptive feature fusion module to fully fuse feature information from different layers and reduce redundant information, thereby solving the problem of low accuracy of lightweight surface defect detection networks.

[0044] (3) The present invention can achieve a better balance between detection accuracy and speed, providing a more applicable solution for embedded devices and scenarios with limited computing power.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic flow chart of a strip steel surface defect detection method based on a lightweight double-reinforced network according to the present invention;

[0047] Figure 2 It is a schematic diagram of the overall process of a strip steel surface defect detection network in a strip steel surface defect detection method based on a lightweight dual-enhancement network of the present invention;

[0048] Figure 3 Schematic diagram of a hybrid distillation strategy for a strip surface defect detection method based on a lightweight dual-reinforcement network according to the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0051] Example

[0052] like Figure 1 As shown, the present invention provides a strip surface defect detection method based on a lightweight double-reinforced network, comprising the following steps:

[0053] S1. Read image samples for strip surface defect detection, process them using data enhancement technology, and then include them in the sample library;

[0054] S2, taking the defect image samples as inputs of the conventional strip surface defect detection network and the lightweight dual enhancement network, respectively, and extracting the feature map of the training image through a feature extraction module stacked by convolutional layers;

[0055] S3, inputting feature mapping information from different levels into a multi-layer coordinated adaptive feature fusion module to ensure efficient fusion and utilization of feature information;

[0056] S4, using a decoupling head to identify and locate feature information;

[0057] S5. The distillation loss function is used to force the lightweight dual reinforcement network to learn the intermediate prompt layer features and the logits output of the classifier of the conventional strip surface defect detection network.

[0058] The multi-layer coordinated adaptive feature fusion module is one of the key components of the network model and plays an internal enhancement role. A hybrid knowledge distillation method is introduced to transfer the knowledge of the conventional strip surface defect detection network to the lightweight dual enhancement network from the outside. By adopting this dual enhancement strategy, a lightweight dual enhancement network that can efficiently detect strip surface defects can be constructed. Compared with mainstream defect detection algorithms, it has a better balance between detection accuracy, speed and lightweight.

[0059] In step S1, the dataset used is the NEU-DET public dataset. It contains 1,800 images, representing six types of defects, 300 images for each defect type, and the size of each image is 200×200 pixels. During the experiment, the dataset is divided into a training set and a test set in a ratio of 7:3.

[0060] Data augmentation techniques are widely used in detection tasks due to their effectiveness. In order to improve detection accuracy, this invention applies data augmentation techniques, such as geometric transformation and contrast enhancement, to better customize the data set for training. After augmentation, the data set contains 3,000 images, of which 2,100 are used for training and 900 are used for testing.

[0061] In step S2, the conventional strip surface defect detection network: 1) uses a C2f module stack consisting of ordinary convolution as the Backbone; 2) introduces a multi-layer coordinated adaptive feature fusion module and a decoupling head to form the Neck part.

[0062] Lightweight dual-enhancement network: 1) FasterNet modules with PConv as the core are stacked to form the backbone of the network; 2) A multi-layer coordinated adaptive feature fusion module and a decoupling head are introduced to form the Neck part; 3) More effective knowledge is extracted from the conventional strip surface defect detection network through a hybrid knowledge distillation strategy.

[0063] In step S3, a multi-layer coordinated adaptive feature fusion module is designed inside the network model to address challenges such as cross-layer cyclic feature information, effective use of semantic and positioning features, and reduction of feature redundancy. The multi-layer coordinated adaptive feature fusion module uses a top-down structure to integrate high-level semantic feature maps. Then, feature maps from different levels are processed through an adaptive fusion (ASFF) module to adjust their information weights. Finally, a bottom-up path is used to fully fuse the positioning information from the lower layer with the semantic information from the higher layer. This "spider web" structure can better fuse features and optimize the balance between accuracy and computational efficiency.

[0064] like Figure 2 As shown in the figure, four levels of feature map channels (P2-P5) are taken as an example. First, the feature maps of the C2-C5 layers in the backbone network are input into the feature fusion module, and then the feature maps are simply circulated by upsampling using the path opposite to feature extraction. Then, the adaptive spatial feature fusion (ASFF) module is introduced to fully fuse and utilize all channel information. Since the operation of ASFF is differentiable, the back propagation algorithm can be used to automatically adjust the weights of feature fusion during the training process, thereby suppressing useless information and amplifying useful information. Finally, the bottom-up path channel is used to enhance high-level semantic features and low-level positioning features.

[0065] For the ASFF module, let the feature of layer n be x n (n can be adjusted according to different network structures. In the example, n∈{2, 3, 4, 5}). For features at different levels, the feature map size needs to be adjusted to the same size. Taking n layers as an example, its adaptive feature fusion is expressed as

[0066]

[0067] in Represents the feature vector at the position (i, j) of the feature map adjusted from different layers to the n layer, Represents the feature vector output along the corresponding channel. They represent the adaptive weights of different layers, which are calculated using the Softmax function. The auxiliary blocks shown in the figure are used to adjust the number of channels in different layers.

[0068] In step S5, the hybrid distillation method adopts a dual-channel strategy, such as Figure 3 As shown in the figure, the outer pipeline represents feature-based distillation, and the inner pipeline represents logit-based distillation. Among them, the outer channel focuses on feature knowledge extraction, while the inner channel focuses more on logits output.

[0069] 1) For feature extraction channels, the currently common probabilistic knowledge transfer method is adopted. First, the feature maps obtained after each channel is processed by the activation function are normalized by the Softmax function to obtain a probability map, and then the KL divergence of the Soft probability map between the student model and the teacher model is minimized, so that the student model learns the important feature representation on the teacher channel.

[0070] The details are as follows. First, we introduce the loss function based on the feature channel distillation method. The teacher network and the student network are represented as T and S respectively, and the corresponding activation maps are represented as y T and S . Then the distillation loss along the channel can be expressed as follows:

[0071]

[0072] Among them, L(·) represents the cross entropy loss, which is used to evaluate the difference in channel distribution between the teacher network and the student network. φ(·) represents the conversion of activation values ​​into probability distribution using the Softmax function, as shown below:

[0073]

[0074] Where c = 1, 2, ···, C represents the channel index, i represents the spatial position index of the channel, and T is the temperature parameter. It can be expressed as follows using KL divergence:

[0075]

[0076] Where x represents a sample. From the above formula, we can see that when the probability knowledge of the student model is similar to that of the teacher model, the KL divergence value can be minimized and more weight can be allocated.

[0077] 2) For the channels focusing on logit output, it is conventionally divided into two parts: classification distillation loss and positioning distillation loss. This method can make up for the problem that feature distillation methods do not pay enough attention to positioning distillation loss.

[0078] ① For the classification distillation loss part of logit distillation, the current common classification distillation method is used, that is, the binary cross entropy loss function is used in conjunction with KL divergence. The classification logistic graph is represented by the binary cross entropy loss function as multiple binary classification graphs, as shown below:

[0079]

[0080] Where, L BCE represents the binary cross entropy loss, represents the total classification distillation loss. And ω=|y T' -y S' | represents the loss weighting coefficient, which is used to measure the importance of sample x.

[0081] ② In view of the good effect of IoU in positioning distillation loss, the present invention introduces a more advanced EIoU loss function for positioning distillation loss. EIOU separates the influencing factors of the aspect ratio of the predicted box and the true box, and then calculates the length and width of the predicted box and the true box respectively. It solves the problem that when the aspect ratio of the predicted box and the true box is the same, w and h cannot be increased or decreased at the same time, resulting in a pair of opposite numbers when calculating the gradient. EIoU consists of three parts: IoU loss, distance loss, and width-height loss (i.e., overlapping area, center point distance, and aspect ratio). The width-height loss is used to minimize the difference in width and height between the predicted target bounding box and the true bounding box, so that it has faster convergence speed and better positioning results. Specifically as follows:

[0082]

[0083] Among them, w c and h c is the width and height of the smallest enclosed area covering the predicted bounding box and the true bounding box. ρ is the Euclidean distance between two points.

[0084] Like the classification loss, the positioning loss function formula after introducing the weight coefficient is:

[0085]

[0086] In the formula, represents the total positioning distillation loss. Therefore, the total distillation loss function based on the logit method is expressed as

[0087]

[0088] Where α and β are hyperparameters used to adjust the loss weights of classification loss and localization loss, respectively. Therefore, the total loss function of hybrid distillation is:

[0089]

[0090] In experimental tests, the default Among them, the feature-based distillation loss is expressed as It uses Softmax to normalize the feature map into a probability map, and then minimizes it using KL divergence. (ii) The logit-based distillation loss is derived from the binary classification cross entropy loss and EIOU positioning loss The classification loss is calculated by converting the logit map into multiple binary classification maps. The localization loss is calculated using the EIOU value between the predicted bounding boxes of the two models.

[0091] Therefore, the present invention adopts the above-mentioned strip surface defect detection method based on a lightweight dual enhancement network. Internally, a lightweight module and an efficient feature fusion module are applied, and externally, hybrid knowledge distillation is used. In this framework, an additional teacher model will be constructed to help the student model to perform hybrid knowledge extraction. The teacher model is called a conventional strip surface defect detection network, and the student model is called a lightweight dual enhancement network. The conventional strip surface defect detection network is composed of a C2f module stack composed of ordinary convolutions as the Backbone, and a multi-layer coordinated adaptive feature fusion module and a decoupling head are introduced to form the Neck part. In order to obtain a detection model that can balance accuracy and speed, the lightweight dual enhancement network: 1) The FasterNet module stack with PConv as the core is used to form the backbone of the network, which can greatly reduce the number of network parameters and improve the detection speed; 2) A multi-layer coordinated adaptive feature fusion module is introduced to fully fuse feature information from different layers and reduce redundant information; 3) A hybrid knowledge distillation strategy is adopted to extract more effective knowledge from the teacher model.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A strip steel surface defect detection method based on a lightweight double-reinforced network, characterized in that: The following steps are involved: S1. Read image samples for strip surface defect detection, process them using data enhancement technology, and then include them in the sample library; S2, taking the defect image samples as inputs of the conventional strip surface defect detection network and the lightweight dual enhancement network, respectively, and extracting the feature map of the training image through a feature extraction module stacked by convolutional layers; S3, inputting feature mapping information from different levels into a multi-layer coordinated adaptive feature fusion module to ensure efficient fusion and utilization of feature information; S4, using a decoupling head to identify and locate feature information; S5. The distillation loss function is used to force the lightweight dual reinforcement network to learn the intermediate prompt layer features and the logits output of the classifier of the conventional strip surface defect detection network.

2. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 1, characterized in that: In step S1, the NEU-DET public dataset is used as the dataset, which contains 1,800 images, representing six types of defects, each defect type has 300 images, the size of each image is 200×200 pixels, and the dataset is divided into a training set and a test set in a ratio of 7:

3.

3. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 1, characterized in that: In step S2, the conventional strip surface defect detection network uses a C2f module stack composed of ordinary convolution as the Backbone, and introduces a multi-layer coordinated adaptive feature fusion module and a decoupling head to form the Neck part.

4. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 1, characterized in that: In step S2, the lightweight dual enhancement network uses the FasterNet module stack with PConv as the core to form the backbone of the network, and introduces a multi-layer coordinated adaptive feature fusion module and a decoupling head to form the Neck part. Finally, a hybrid knowledge distillation strategy is used to extract more effective knowledge from the conventional strip surface defect detection network.

5. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 1, characterized in that: In step S3, the multi-layer coordinated adaptive feature fusion module adopts a top-down structure to integrate high-level semantic feature maps, processes the feature maps from different levels through the adaptive fusion ASFF module to adjust the weights, and then adopts a bottom-up path to completely fuse the positioning information from the lower layer with the semantic information from the higher layer.

6. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 5, characterized in that: In the adaptive fusion ASFF module, let the feature of layer n be x n , n∈{2,3,4,5}, its adaptive feature fusion Expressed as in Represents the feature vector at the position (i, j) of the feature map adjusted from different layers to the n layer, Represents the feature vector output along the corresponding channel; They represent the adaptive weights of different levels, which are calculated using the Softmax function.

7. The method for detecting surface defects of strip steel based on a lightweight double-reinforced network according to claim 1, characterized in that: In step S5, the hybrid distillation method adopts a dual-channel strategy, which is mainly composed of two distillation losses based on the feature channel and the logit output channel, as shown below: S51, the teacher network and the student network are represented as T and S respectively, and the corresponding activation maps are represented as y T and S , then the distillation loss based on the feature channel can be expressed as follows: Among them, L(·) represents the cross entropy loss, which is used to evaluate the difference in channel distribution between the teacher network and the student network, and φ(·) represents the conversion of activation values ​​into probability distribution using the Softmax function, as shown below: Where c = 1, 2, ···, C represents the channel index, i represents the spatial position index of the channel, and T is the temperature parameter, which is expressed as follows using KL divergence: Where x represents a sample. From the above formula, we can see that when the probability knowledge of the student model is similar to that of the teacher model, the KL divergence value can be minimized. S52. The distillation loss based on logit output is divided into two parts: classification distillation loss and positioning distillation loss. The classification distillation loss part uses the binary cross entropy loss function in conjunction with the KL divergence, as shown below: Where, L BCE represents the binary cross entropy loss, represents the total classification distillation loss, ω=|y T '-y S '| represents the loss weight coefficient, which is used to measure the importance of sample x; The EIoU loss function is used to locate the distillation loss. EIoU consists of three parts: IoU loss, distance loss, and width and height loss, as shown below: Where A and B are the areas of the predicted bounding box and the true bounding box respectively, |A∩B| represents the intersection area of ​​the two areas, and |A∪B| represents the union area of ​​the two areas; Among them, w c and h c is the width and height of the minimum closed area covering the predicted bounding box and the true bounding box, and ρ is the Euclidean distance between the two points; Like the classification loss, the positioning loss function formula after introducing the weight coefficient is: In the formula, represents the total positioning distillation loss; therefore, the total distillation loss function based on the logit method is expressed as In the formula α and β are hyperparameters used to adjust the loss weights of classification loss and localization loss respectively; Therefore, the total loss function of hybrid distillation is: Among them, the feature-based distillation loss is expressed as It uses Softmax to normalize the feature map into a probability map, and then minimizes the KL divergence; the logit-based distillation loss is obtained by the binary classification cross entropy loss and EIOU positioning loss The classification loss is calculated by converting the logit map into multiple binary classification maps. The localization loss is calculated using the EIOU value between the predicted bounding boxes of the two models.

Images