A method and apparatus for detecting defects on an industrial surface

Abstract

The application discloses a model compression method and device based on channel pruning. According to the structural similarity and detail richness of feature maps (also called channels), feature maps that do not meet preset conditions are removed to obtain a pruning model to be trained. A teacher model to be trained is trained to obtain a first teacher model focusing on representation learning and a second teacher model focusing on classification learning. The two teacher models are knowledge-fused to guide the training of the pruning model to be trained, and finally used for compressing the model. The importance of channels is measured by evaluating the structural similarity and detail richness of feature maps, which can effectively capture the features of defect categories, thereby helping to alleviate the class imbalance problem. In addition, the knowledge fusion of the two teacher models can take into account the representation learning and classification learning of the pruning model, and also improves the class imbalance problem.

Description

Technical Field

[0001] This application relates to the field of industrial surface defect detection technology, and in particular to a method and apparatus for industrial surface defect detection. Background Technology

[0002] Surface defect detection technology is a crucial method for addressing the impact of surface defects on product quality and has been widely applied in industries such as steel and photovoltaics. Deep learning-based methods can identify defects by automatically extracting features hidden in data and learning decision rules. Among these, automatic detection frameworks based on Convolutional Neural Networks (CNNs) perform well in surface defect detection tasks, but they are resource-intensive and memory-intensive, making them difficult to deploy in resource-constrained environments. To compress CNN models, structured pruning can be employed, which involves removing relatively unimportant channels. Therefore, the key to implementing structured pruning lies in evaluating the importance of each channel.

[0003] In existing technologies, methods for evaluating channel importance assume that different classes have similar proportions in the collected dataset. However, in industrial surface defect detection tasks, the number of normal class samples is usually much greater than the number of defect class samples, leading to a severe class imbalance problem. Therefore, how to reduce class imbalance while compressing the model has become an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method and apparatus for detecting industrial surface defects, which reduces class imbalance problems while compressing the model.

[0005] This application discloses a method for detecting industrial surface defects, the method comprising: Based on the structural similarity and detail richness of the feature maps, feature maps that do not meet the preset conditions are removed to obtain the pruned model to be trained. The teacher model to be trained is trained to obtain a first teacher model and a second teacher model; the first teacher model focuses on representation learning; the second teacher model focuses on classification learning. The first teacher model and the second teacher model are fused to obtain a new teacher model. The teacher model is used to guide the training of the pruning model to be trained, and the training results are used as the pruning model. The model is compressed using the pruning model described above.

[0006] Optionally, in removing feature maps that do not meet preset conditions based on the structural similarity and detail richness of the feature maps, the method further includes: obtaining the structural similarity value and detail richness of the feature maps.

[0007] Optionally, obtaining the detail richness of the feature map includes: Calculate the number of non-zero singular values ​​in the feature map; The feature map detail richness is evaluated based on the number of non-zero singular values.

[0008] Optionally, removing feature maps that do not meet preset conditions based on structural similarity and detail richness includes: Based on the structural similarity values, the feature maps are divided into multiple similarity classes using a clustering method; In each similarity class, determine whether the detail richness of each feature map is greater than a preset value; If so, the feature map is determined to meet the preset conditions, and the feature map is retained; If not, the feature map is determined to not meet the preset conditions, and the feature map is removed.

[0009] Optionally, the teacher model to be trained is trained to obtain a first teacher model and a second teacher model, including: The first teacher model is obtained by training the teacher model to be trained using the instance balanced sampling method. The teacher model is trained using a class-balanced sampling method to obtain the second teacher model.

[0010] Based on the above-mentioned method for detecting industrial surface defects, this application also discloses an apparatus for detecting industrial surface defects, including: a pruning unit, a teacher model training unit, a knowledge fusion unit, a training unit, and a compression unit; The pruning unit is used to remove feature maps that do not meet the preset conditions based on the structural similarity and detail richness of the feature maps, so as to obtain the pruned model to be trained. The teacher model training unit is used to train the teacher model to be trained, resulting in a first teacher model and a second teacher model; the first teacher model focuses on representation learning; the second teacher model focuses on classification learning. The knowledge fusion unit is used to fuse the knowledge of the first teacher model and the second teacher model to obtain a teacher model. The training unit is used to guide the training of the pruning model to be trained using the teacher model, and obtain the training result as the pruning model. The compression unit is used to compress the model using the pruning model.

[0011] Optionally, the apparatus further includes a preprocessing unit for obtaining structural similarity values ​​and detail richness of the feature map.

[0012] Optionally, the preprocessing unit includes: A computational subunit is used to calculate the number of non-zero singular values ​​in the feature map; An evaluation subunit is used to evaluate the detail richness of the feature map based on the number of non-zero singular values.

[0013] Optionally, the pruning unit includes: A clustering subunit is used to divide the feature map into multiple similar classes based on the structural similarity value; The judgment subunit is used to determine whether the detail richness of each feature map is greater than a preset value in each similarity class; A retention subunit is used to determine if the feature map meets preset conditions and to retain the feature map; The sub-unit is removed to determine if the feature map does not meet the preset conditions and to remove the feature map.

[0014] Optionally, the teacher model training unit includes: The first teacher model obtains a sub-unit, which is used to train the teacher model to be trained using the instance balanced sampling method, to obtain the first teacher model; The second teacher model acquisition sub-unit is used to train the teacher model using the class balanced sampling method to obtain the second teacher model.

[0015] This application discloses a method and apparatus for detecting industrial surface defects. Based on the structural similarity and detail richness of feature maps (also called channels), feature maps that do not meet preset conditions are removed, resulting in a pruning model to be trained. A teacher model is then trained to obtain a first teacher model focusing on representation learning and a second teacher model focusing on classification learning. The knowledge from these two models is then fused to guide the training of the pruning model, which is ultimately used to compress the model. By evaluating the structural similarity and detail richness of feature maps to measure the importance of channels, the characteristics of defect categories can be effectively captured, thus helping to alleviate the class imbalance problem. Furthermore, fusing the knowledge from the two teacher models can balance representation learning and classification learning in the pruning model, further improving the class imbalance problem. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of an industrial surface defect detection method disclosed in an embodiment of this application; Figure 2This is a schematic diagram comparing the results of training models using different sampling methods as disclosed in the embodiments of this application; Figure 3 This is a schematic flowchart of another method for detecting industrial surface defects disclosed in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an industrial surface defect detection device disclosed in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Example 1: This application discloses a method for detecting industrial surface defects.

[0020] For details, please refer to Figure 1 The method for detecting industrial surface defects disclosed in this embodiment includes the following steps: Step 101: Based on the structural similarity and detail richness of the feature maps, remove feature maps that do not meet the preset conditions to obtain the pruned model to be trained.

[0021] In the method described in this embodiment, when obtaining the structural similarity and detail richness of feature maps, as an optional method, the structural similarity value is calculated using the following formula. : (1) in, and Represents any two feature maps in the l-th convolutional layer. Let represent the Schatten p-norm of the eigenmap matrix (p∈(0,+∞)).

[0022] For feature map o l Its corresponding Schatten p-norm can be expressed as follows: (2) Where, σ i Let x = min(m, n) represent the i-th singular value of the feature map matrix, where m and n represent the height and width of the feature map, respectively.

[0023] As an alternative method, singular values ​​can be calculated using singular value decomposition. l Its defining formula is: (3) Among them, u i and v i σ i The left and right singular vectors are given, and T represents the transpose of the matrix.

[0024] In the method described in this embodiment, the Schatten-p norm is used to calculate the structural similarity value. This norm primarily reflects the distribution of a few large singular values, which are related to structural information, thus allowing for the evaluation of the structural similarity of the feature maps. For example, when the difference between the structural similarity values ​​of one feature map and another is less than a preset difference, it can be determined that the two feature maps belong to the same category. The example of the application of the structural similarity value here is for illustrative purposes only; no specific limitations are placed on its application method. The goal is simply to evaluate structural similarity based on the structural similarity value.

[0025] In the method described in this embodiment, when obtaining the detail richness of a feature map, as an optional method, the number of non-zero singular values ​​is used to estimate the detail richness of a single feature map, as shown in the following formula: (4) Where x represents the total number of singular values ​​and n represents the number of non-zero singular values.

[0026] Non-zero singular values ​​can reflect the intensity information of an image, and therefore can also be used to evaluate the detail richness of a feature map. For example, when the number of non-zero singular values ​​in a feature map exceeds a preset number, the detail richness of this feature map is judged to be high or a specific value. The examples of methods for evaluating detail richness and their representations here are only for ease of understanding; no specific limitations are made on the evaluation methods and representations of detail richness, as long as they can be evaluated based on the number of non-zero singular values.

[0027] In the method described in this embodiment, feature maps can be divided into multiple similarity classes based on structural similarity values ​​using a clustering method. As an optional method, feature-related guided clustering is employed: first, the distance matrix of the structural similarity values ​​of the l-th convolutional layer is calculated, and then... Initialize the hierarchical structure so that each feature map is an independent hierarchical structure. Where, c out This represents the number of channels in the l-th convolutional layer. Layers are merged when the average structural similarity value of their clusters is less than a threshold parameter t.

[0028] Feature-related clustering guides the grouping of feature maps with similar structures into the same level. Furthermore, it can classify background and defect features into different levels based on structural similarity scores, helping to eliminate interference from background features.

[0029] In the method described in this embodiment, as an optional approach to determining whether a feature map meets preset conditions, feature maps with less detailed information are not considered to meet the preset conditions (and are not important) when the structures are similar. Specifically, the detail richness of each feature map in each similarity class can be determined to be greater than a preset value. If it is, the feature map is considered to meet the preset conditions and is retained. If not, the feature map is considered to not meet the preset conditions and is removed.

[0030] Step 102: Train the teacher model to be trained to obtain the first teacher model and the second teacher model.

[0031] In the method described in this embodiment, the first teacher model focuses on representation learning, and the second teacher model focuses on classification learning. As an optional method, the first teacher model can be obtained by training the teacher model to be trained using instance balanced sampling, and the second teacher model can be obtained by training the teacher model using class balanced sampling.

[0032] In the method described in this embodiment, the model learning process is divided into representation learning and classification learning. Instance-balanced samplers and class-balanced samplers are used as two types of samplers, and corresponding learning strategies are constructed to train the teacher model.

[0033] like Figure 2 As shown, the F1 scores obtained on the ELPV-DC dataset using the VGG-16 model and the RepVGG-A0 model with different training strategies are illustrated. The vertical axis represents the classification learning strategy, and the horizontal axis represents the representation learning strategy. For example, 0.9054 is obtained by learning representations using the instance balanced sampler 201, then fixing the parameters of the representation layer and training the classifier using the class balanced sampler 202. In the method described in this embodiment, the vertical comparison shows the superiority of different classification learning strategies for the same representation learning strategy, and the horizontal comparison shows the superiority of different representation learning strategies. Therefore, the instance balanced sampler 201 can promote representation learning, and the class balanced sampler 202 can promote classification learning.

[0034] Step 103: Perform knowledge fusion on the first teacher model and the second teacher model to obtain a teacher model.

[0035] In the method described in this embodiment, as an optional approach, knowledge from a first teacher model trained with an instance-balanced sampler and a second teacher model trained with a class-balanced sampler are fused. Specifically, an adaptive knowledge fusion function λ is designed to fuse knowledge from the first and second teacher models, defined as follows: (5) Among them, T and T max These represent the current training cycle and the total training cycle, respectively.

[0036] Step 104: Use the teacher model to guide the training of the pruning model to be trained, and obtain the training results as the pruning model.

[0037] In the method described in this embodiment, logical knowledge is transferred from the teacher model using the KL divergence loss function.

[0038] The total loss L of the pruning model is as follows: (6) Among them, L RKD L represents the information conveyed using the first teacher model. CKD This represents the information conveyed using the second teacher model.

[0039] (7) (8) Where τ represents the temperature parameter, C represents the number of categories, and σ(·) represents the softmax function. , and These represent the outputs of the first teacher model, the second teacher model, and the student model, respectively.

[0040] In the method described in this embodiment, as the training cycle increases, the adaptive knowledge fusion function will gradually decrease, thereby shifting the learning focus from representation ability to defect class identification, which helps to reduce class imbalance.

[0041] Step 105: Compress the model using the pruning model.

[0042] The method described in this embodiment selects feature maps that meet preset conditions (important) based on structural similarity and detail richness. This allows for the acquisition of features related to defect categories, helping to alleviate class imbalance problems. Simultaneously, this embodiment proposes a knowledge fusion method that integrates knowledge from two teacher models to balance representation learning and classification learning. Furthermore, the method described in this embodiment is a one-time pruning strategy that can be used after pruning all layers of the model to fine-tune the pruning process. Compared to iterative pruning, the method described in this embodiment improves pruning efficiency.

[0043] Example 2: This application discloses another method for industrial surface defect detection. Please refer to [link / reference]. Figure 3 The method described in this embodiment is for the pruning model to be trained.

[0044] Step 301: Obtain the structural similarity and detail richness of the feature maps.

[0045] Step 302: Classify the feature maps based on structural similarity.

[0046] Step 303: Based on the detail richness of the feature maps in each class, remove feature maps that do not meet the preset conditions to obtain the pruned model to be trained.

[0047] Step 304: Use the instance balanced sampler to train the teacher model to be trained to obtain the first teacher model.

[0048] Step 305: Use a balanced sampler to train the teacher model to be trained to obtain the second teacher model.

[0049] Step 306: Integrate knowledge from the first teacher model and the second teacher model.

[0050] Step 307: Use the fused model as the teacher model to train the pruning model to be trained.

[0051] The method described in this embodiment measures channel importance by evaluating the structural similarity and detail richness of feature maps, effectively capturing features of defect categories and thus helping to alleviate class imbalance. Furthermore, fusing knowledge from two teacher models balances representation learning and classification learning in the pruning model, further improving class imbalance.

[0052] The method described in this embodiment can be used for compressed surface defect detection models. The universality and superiority of the method described in this embodiment can be demonstrated by conducting experiments on the ELPV-DC dataset and the NEU-CLS dataset.

[0053] As an alternative approach, the ELPV-DC dataset consists of 2,624 electroluminescent images containing various photovoltaic cell defects, such as cracks, short-circuited cells, weld faults, and interconnections. The NEU-CLS dataset consists of 1,800 grayscale images from six types of steel defects, including cracks, scratches, paths, inclusions, pitting, and rolling defects.

[0054] The method described in this embodiment can be tested using mainstream backbone networks, including the VGG-16 network model implemented using a common architecture, the RepVGG-A0 network model implemented using a reparameterized architecture, the ResNet-50 network model implemented using residual blocks, and the MobileNetV2 network model implemented using depthwise separable convolutions.

[0055] The pruning results of the VGG-16 model on the ELPV-DC dataset are shown in Table 1: Table 1. Pruning results of VGG-16 on the ELPV-DC dataset.

[0056] In the table, CFP refers to the method described in this embodiment. This is a method for fine-tuning the model using image ground truth labels after applying pruning strategies from CFP.

[0057] As shown in Table 1, without comparison with CFP, CFP Compared to other methods, it reduces the number of parameters and computational cost without compromising accuracy. With similar parameter and computational costs, CFP... It boasts higher accuracy and F1 score compared to other methods. For example, compared to CFP... Compared to the White Box, which has a similar level of accuracy, CFP This reduces the number of parameters and computational load. The CFP in Table 1... Compared with CFP, it can be found that CFP has a higher accuracy rate than CFP. The increase of 0.96% indicates that the knowledge fusion step in CFP can mitigate the impact of class imbalance.

[0058] The pruning results of the RESNET-50 model on the NEU-CLS dataset are shown in Table 2: Table 2. Pruning results of RESNET-50 on the NEU-CLS dataset.

[0059] As can be seen, for the ResNet-50 model on the NEU-CLS dataset, CFP outperforms other methods in terms of computational cost and parameter count while experiencing minimal accuracy loss. Compared to HRank and CHIP, CFP can reduce the number of parameters and computational cost by a greater margin while maintaining similar accuracy. Therefore, CFP can effectively compress models with residual blocks.

[0060] Furthermore, by evaluating the prediction time of CFP on VGG-16, RepVGG-A0, DDDN, and ResNet-50 models on the ELPV-DC dataset, its effectiveness in practical applications can be demonstrated. Specifically, the acceleration effects of some models are shown in Table 3: Table 3. Acceleration effect of CFP on the ELPV-DC dataset on NVIDIA RTX 3090 graphics card

[0061] As can be seen, CFP achieved speedups of 3.06x and 1.53x for VGG-16 and RepVGG-A0, respectively. Furthermore, CFP achieved a 1.78x speedup for the lightweight DDDN network. Therefore, CFP can significantly accelerate model prediction, which is helpful for achieving online defect detection in industrial production scenarios.

[0062] Based on the method for detecting industrial surface defects disclosed in the above embodiments, this embodiment correspondingly discloses an apparatus for detecting industrial surface defects. Please refer to... Figure 4 The device for detecting industrial surface defects includes: a pruning unit 401, a teacher model training unit 402, a knowledge fusion unit 403, a training unit 404, and a compression unit 405. The pruning unit 401 is used to remove feature maps that do not meet the preset conditions based on the structural similarity and detail richness of the feature maps, so as to obtain the pruning model to be trained. The teacher model training unit 402 is used to train the teacher model to be trained to obtain a first teacher model and a second teacher model; the first teacher model focuses on representation learning; the second teacher model focuses on classification learning. The knowledge fusion unit 403 is used to perform knowledge fusion on the first teacher model and the second teacher model to obtain a teacher model; The training unit 404 is used to guide the training of the pruning model to be trained using the teacher model, and obtain the training result as the pruning model. The compression unit 405 is used to compress the model using the pruning model.

[0063] Optionally, the apparatus further includes a preprocessing unit for obtaining structural similarity values ​​and detail richness of the feature map.

[0064] Optionally, the preprocessing unit includes: A computational subunit is used to calculate the number of non-zero singular values ​​in the feature map; An evaluation subunit is used to evaluate the detail richness of the feature map based on the number of non-zero singular values.

[0065] Optionally, the pruning unit 401 includes: A clustering subunit is used to divide the feature map into multiple similar classes based on the structural similarity value; The judgment subunit is used to determine whether the detail richness of each feature map is greater than a preset value in each similarity class; A retention subunit is used to determine if the feature map meets preset conditions and to retain the feature map; The sub-unit is removed to determine if the feature map does not meet the preset conditions and to remove the feature map.

[0066] Optionally, the teacher model training unit 402 includes: The first teacher model obtains a sub-unit, which is used to train the teacher model to be trained using the instance balanced sampling method, to obtain the first teacher model; The second teacher model acquisition sub-unit is used to train the teacher model using the class balanced sampling method to obtain the second teacher model.

[0067] The embodiments in this specification are described in a progressive manner. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant details can be found in the method section.

[0068] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0070] The features described in the embodiments of this specification can be substituted for or combined with each other, so that those skilled in the art can implement or use this application.

[0071] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of industrial surface defect detection, characterized in that, include: Based on the structural similarity and detail richness of the feature maps, feature maps that do not meet the preset conditions are removed to obtain the pruned model to be trained. The teacher model to be trained is trained to obtain the first teacher model and the second teacher model; The first teacher model focuses on representation learning; The second teacher model focuses on classification learning; The first teacher model and the second teacher model are fused to obtain a new teacher model. The teacher model is used to guide the training of the pruning model to be trained, and the training results are used as the pruning model. The pruning model is used to compress the industrial surface defect detection model; The compressed industrial surface defect detection model is used to perform industrial surface defect detection tasks.

2. The method of claim 1, wherein, The method further includes removing feature maps that do not meet preset conditions based on the structural similarity and detail richness of the feature maps.

3. The method of claim 2, wherein, Obtaining the detail richness of the feature map includes: Calculate the number of non-zero singular values ​​in the feature map; The feature map detail richness is evaluated based on the number of non-zero singular values.

4. The method of claim 2, wherein, The step of removing feature maps that do not meet preset conditions based on structural similarity and detail richness includes: Based on the structural similarity values, the feature maps are divided into multiple similarity classes using a clustering method; In each similarity class, determine whether the detail richness of each feature map is greater than a preset value; If so, the feature map is determined to meet the preset conditions, and the feature map is retained; If not, the feature map is determined to not meet the preset conditions, and the feature map is removed.

5. The method of claim 1, wherein, The teacher model to be trained is then trained to obtain the first teacher model and the second teacher model, including: The first teacher model is obtained by training the teacher model to be trained using the instance balanced sampling method. The teacher model is trained using a class-balanced sampling method to obtain the second teacher model.

6. An apparatus for industrial surface defect detection, characterized in that, include: The unit includes a pruning unit, a teacher model training unit, a knowledge integration unit, a training unit, a compression unit, and a testing unit. The pruning unit is used to remove feature maps that do not meet the preset conditions based on the structural similarity and detail richness of the feature maps, so as to obtain the pruned model to be trained. The teacher model training unit is used to train the teacher model to be trained, to obtain a first teacher model and a second teacher model; the first teacher model focuses on representation learning. The second teacher model focuses on classification learning; The knowledge fusion unit is used to fuse the knowledge of the first teacher model and the second teacher model to obtain a teacher model. The training unit is used to guide the training of the pruning model to be trained using the teacher model, and obtain the training result as the pruning model. The compression unit is used to compress the industrial surface defect detection model using the pruning model; The detection unit is used to perform industrial surface defect detection tasks using a compressed industrial surface defect detection model.

7. The apparatus of claim 6, wherein, The device further includes a preprocessing unit for obtaining the structural similarity value and detail richness of the feature map.

8. The apparatus of claim 7, wherein, The preprocessing unit includes: A computational subunit is used to calculate the number of non-zero singular values ​​in the feature map; An evaluation subunit is used to evaluate the detail richness of the feature map based on the number of non-zero singular values.

9. The apparatus according to claim 7, characterized in that, The pruning unit includes: A clustering subunit is used to divide the feature map into multiple similar classes based on the structural similarity value; The judgment subunit is used to determine whether the detail richness of each feature map is greater than a preset value in each similarity class; A retention subunit is used to determine if the feature map meets preset conditions and to retain the feature map; The sub-unit is removed to determine if the feature map does not meet the preset conditions and to remove the feature map.

10. The apparatus of claim 6, wherein, The teacher model training unit includes: The first teacher model obtains a sub-unit, which is used to train the teacher model to be trained using the instance balanced sampling method, to obtain the first teacher model; The second teacher model acquisition sub-unit is used to train the teacher model using the class balanced sampling method to obtain the second teacher model.

Images