Lightweight defect detection method based on hybrid multi-scale knowledge distillation

Abstract

Disclosed in the present invention is a lightweight defect detection method based on hybrid multi-scale knowledge distillation. The method comprises: constructing a dataset; constructing a teacher network model and a lightweight student network model; using the dataset to train the teacher network model, and saving a weight file of the trained teacher network model; and loading into the teacher network model the saved weight file of the teacher network model, inputting defect images in the dataset into the teacher network model and the student network model to respectively obtain first multi-scale features and second multi-scale features, respectively inputting the first multi-scale features and the second multi-scale features into a cascaded knowledge blending module to obtain final deeply fused first multi-scale features and final deeply fused second multi-scale features, then calculating a hybrid multi-scale knowledge loss, and in combination with the prediction loss of the student network model, using a backpropagation algorithm to update network parameters, so as to obtain a trained lightweight student network model for implementing defect detection of intelligent manufacturing products. The cognitive ability and recognition performance for defects of different scales are improved.

Description

Lightweight defect detection method based on mixed multi-scale knowledge distillation

[0001] The present application claims priority to the Chinese patent application filed on May 11, 2024, with the Chinese Patent Office, the number of which is 2024105799809, and the title of which is "Lightweight defect detection method based on mixed multi-scale knowledge distillation", the whole content or part of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application belongs to the field of defect detection technology in intelligent manufacturing, and particularly relates to a lightweight defect detection method based on mixed multi-scale knowledge distillation. BACKGROUND

[0003] With the rapid development of the intelligent manufacturing industry, using deep learning technology to train deep detection models for automatic detection of product defects in the manufacturing process has become a trend. Defect detection helps to improve product quality and production efficiency, and realizes the intelligentization and high efficiency of the manufacturing process.

[0004] In the intelligent manufacturing process, there are often resource constraints, such as mobile devices, embedded systems or edge computing platforms, etc. In these environments, traditional deep learning models are often too complex, and the large number of parameters makes the large model take a long time to detect product defects in the intelligent manufacturing process, which cannot meet the requirements of real-time applications. Lightweight models refer to reducing the number of parameters and computational complexity of the model, reducing the storage requirements and computational complexity of the model while maintaining high model performance. Therefore, using lightweight models for defect detection can realize efficient model inference under limited resource conditions, improve the inference speed of the model, and meet the real-time requirements of the intelligent manufacturing industry.

[0005] Using lightweight models for defect detection has the advantages of improving resource efficiency, enhancing real-time performance and improving product quality. A lightweight defect detection method based on mixed multi-scale knowledge distillation is adopted to promote the development of the intelligent manufacturing industry and realize efficient, accurate and sustainable intelligent manufacturing production. SUMMARY

[0006] To solve the above technical problems, the present application provides a lightweight defect detection method based on mixed multi-scale knowledge distillation.

[0007] The technical solution adopted by the present application to solve its technical problems is:

[0008] A lightweight defect detection method based on mixed multi-scale knowledge distillation, the method comprising the following steps:

[0009] S100: photographing the intelligent manufacturing product, pre-processing the photographed photo, labeling using a labeling software, and constructing a data set for network training;

[0010] S200: constructing a teacher network model and a lightweight student network model for intelligent manufacturing defect detection;

[0011] S300: training the teacher network model using the data set, and saving the trained teacher network model weight file;

[0012] S400: loading the saved teacher network model weight file into the teacher network model, inputting the defect images in the data set into the teacher network model and the student network model to obtain first multi-scale features and second multi-scale features respectively, inputting the first multi-scale features and the second multi-scale features into multiple dilated branches respectively to obtain scale perception enhanced first multi-scale features and scale perception enhanced second multi-scale features, inputting the scale perception enhanced first multi-scale features and the scale perception enhanced second multi-scale features into a fusion feature module respectively to obtain depth fused first multi-scale features and depth fused second multi-scale features, respectively passing through a preset number of dilated branches and a feature fusion module to obtain final depth fused first multi-scale features and final depth fused second multi-scale features, calculating a hybrid multi-scale knowledge loss according to the final depth fused first multi-scale features and the final depth fused second multi-scale features, updating parameters of the student network model using a back propagation algorithm combined with a total loss composed of the hybrid multi-scale knowledge loss and a prediction loss of the student network model, and obtaining a trained lightweight student network model;

[0013] S500: deploying and applying the trained lightweight student network model to an autonomous mobile robot to realize dynamic and real-time defect detection of intelligent manufacturing products.

[0014] Preferably, the teacher network model in S200 includes a residual network with a depth of 101 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for detection result prediction; wherein the feature pyramid module first constructs multi-scale features with different sizes and the same number of channels using a convolution layer, and then realizes top-down feature fusion using bilinear interpolation to realize the fusion of semantic features and pixel-level features; the detection head module first encodes the fused multi-scale features using four convolution layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolution layers respectively.

[0015] Preferably, the lightweight convolutional neural network model in S200 includes a residual network with a depth of 18 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for detection result prediction; wherein the feature pyramid module first constructs multi-scale features with different scale sizes and the same number of channels using convolutional layers, and then realizes top-down feature fusion using bilinear interpolation to realize the fusion of semantic features and pixel-level features; the detection head module first encodes the fused multi-scale features using four convolutional layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolutional layers respectively.

[0016] Preferably, the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature obtained by inputting the first multi-scale feature and the second multi-scale feature into the multi-dilated branch in S400 include:

[0017] S410: input the first multi-scale feature and the second multi-scale feature into convolutional layer branches with different dilation rates d, and then obtain the receptive field enhanced first multi-scale feature and the receptive field enhanced second multi-scale feature; wherein d = 1, 2, 3;

[0018] S420: splice the receptive field enhanced first multi-scale feature and the first multi-scale feature, and splice the receptive field enhanced second multi-scale feature and the second multi-scale feature to obtain new multi-scale features Or

[0019] S430: reduce the dimension of the new multi-scale feature through a 1x1 convolutional layer to obtain a scale consistent with the original multi-scale feature dimension, and obtain the perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature.

[0020] Preferably, the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature obtained by inputting the first multi-scale feature and the second multi-scale feature into the multi-dilated branch in S400 include:

[0021] wherein i represents the feature of the ith scale, T represents the teacher detection model, L represents the lightweight defect detection model, conv di represents a 3x3 convolution with different dilation rates, C represents splicing along the channel dimension, represents the first multi-scale feature, represents the second multi-scale feature, conv1 represents a 1x1 convolutional layer for dimension reduction, represents the scale perception enhanced first multi-scale feature, represents the scale perception enhanced second multi-scale feature.

[0022] Preferably, the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature in S400 are input into the fusion feature module respectively to obtain the depth fused first multi-scale feature and the depth fused second multi-scale feature, comprising:

[0023] S440: using bilinear interpolation technology and 3x3 convolution to interpolate the features adjacent to the scale perception enhanced first multi-scale feature to obtain a scale feature with the same resolution size as the scale perception enhanced first multi-scale feature; using bilinear interpolation technology and 3x3 convolution to interpolate the features adjacent to the scale perception enhanced second multi-scale feature to obtain a scale feature with the same resolution size as the scale perception enhanced second multi-scale feature;

[0024] S450: using deformable convolution to extract the defect scale related features of the scale perception enhanced first multi-scale feature and the scale feature with the same resolution size as the scale perception enhanced first multi-scale feature, and extract the defect scale related features of the scale perception enhanced second multi-scale feature and the scale feature with the same resolution size as the scale perception enhanced first multi-scale feature;

[0025] S460: adding the extracted defect scale related features respectively to realize the fusion between adjacent scale features, and obtain the fused first multi-scale feature and the fused second multi-scale feature;

[0026] S470: processing the fused first multi-scale feature and the fused second multi-scale feature respectively through integrated normalization technology, and combining an activation function to ensure the nonlinearity of data, and obtaining the depth fused first multi-scale feature and the depth fused second multi-scale feature.

[0027] Preferably, S440 is specifically:

[0028] Wherein, F bi represents bilinear interpolation technology, conv3 represents a 3x3 convolution layer with a step of 2, and represent adjacent scale features with the same resolution size as , and represent adjacent scale features with the same resolution size as ;

[0029] S450 is specifically:

[0030] wherein dconv denotes deformable convolution to extract defect scale related features, denotes and the defect scale related features extracted by deformable convolution, denotes and the defect scale related features extracted by deformable convolution.

[0031] Preferably, S460 is specifically:

[0032] wherein, and denote the fused first multi-scale features and the fused second multi-scale features;

[0033] S470 is specifically:

[0034] wherein ReLU denotes an activation function, IBN denotes an integrated normalization function, and denote the depth fused first multi-scale features and the depth fused second multi-scale features, respectively.

[0035] Preferably, the mixed multi-scale knowledge loss is calculated in S400 according to the final depth fused first multi-scale features and the final depth fused second multi-scale features, comprising:

[0036] wherein Loss ms denotes the mixed multi-scale knowledge loss, n denotes a feature sample amount, and denote the final depth fused first multi-scale features and the final depth fused second multi-scale features, respectively.

[0037] Preferably, the total loss in S400 is specifically: Loss = λ ms Loss ms + λ cls Loss cls + λ reg Loss reg

[0038] wherein λ ms , λ cls and λ reg adjust the balance between Loss ms , Loss cls and Loss reg , respectively; Loss cls = -y(1-p)γ log(p) - (1 - y)p γ log(1 - p)

[0039] where y represents the true label of the defect category, p represents the predicted category, and γ is a hyperparameter for adjusting the classification loss;

[0040] where IoU represents the intersection over union between the real label and the predicted box, C represents the minimum closed shape, A c represents the area of C, and U represents the sum of the areas of A and B.

[0041] The above-mentioned lightweight defect detection method based on mixed multi-scale knowledge distillation transfers the knowledge learned by the teacher network model to the lightweight student network model to complete knowledge transfer. The cascaded knowledge mixing module obtains deep mixed scale knowledge by cascading multiple multi-dilation convolution branches and feature mixing modules. By transferring this mixed multi-scale knowledge, the knowledge distillation process is improved, which helps to improve the cognitive ability of the lightweight defect detection student network model for defects of different scales, and thus improves the recognition performance of defects. Using a lightweight model for defect detection has the advantages of improving resource efficiency, enhancing real-time performance, and improving product quality, promoting the development of intelligent manufacturing, and realizing efficient, accurate, and sustainable intelligent manufacturing production. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of a lightweight defect detection method based on mixed multi-scale knowledge distillation according to an embodiment of the present application;

[0043] Figure 2 is a framework diagram of a lightweight defect detection method based on mixed multi-scale knowledge distillation according to an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of a multi-dilation convolution branch structure according to an embodiment of the present application;

[0045] Figure 4 is a feature fusion module structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings.

[0047] In one embodiment, as shown in Figure 1, a lightweight defect detection method based on mixed multi-scale knowledge distillation, the method comprises the following steps:

[0048] S100: Photographing the intelligent manufacturing product and pre-processing the photographed photo, using a labeling software to label and constructing a data set for network training.

[0049] Specifically, first, a high-definition industrial camera is used to take pictures of defects generated in the intelligent manufacturing process to obtain a set of original image data; then, a special labeling software Labelme is used to label the defect parts in the pictures to obtain a dataset for lightweight defect network training; finally, data enhancement is performed on the labeled dataset by using data enhancement methods such as flipping and scaling, and the dataset is divided into a training set and a test set according to a specific proportion and data category distribution.

[0050] S200: Construct a teacher network model and a lightweight student network model for intelligent manufacturing defect detection.

[0051] In one embodiment, the teacher network model in S200 includes a residual network with a depth of 101 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for defect result prediction; wherein the feature pyramid module first constructs multi-scale features with different scale sizes and the same number of channels using convolution layers, and then realizes top-down feature fusion using bilinear interpolation to realize the fusion of semantic features and pixel-level features, and the detection head module first encodes the fused multi-scale features using four convolution layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolution layers.

[0052] Specifically, the teacher network model can use a deeper and more complex network model to learn more complex defect-related knowledge, and then transfer the knowledge to the lightweight student network model through mixed multi-scale knowledge distillation. First, the teacher network model uses a residual network with a depth of 101 layers for multi-scale feature extraction of defects, which improves the network's ability to represent defects of different sizes; then, a five-layer feature pyramid structure is used for multi-scale feature fusion to ensure that the semantic information of the defect and the pixel-level information of the defect are deeply fused; finally, a prediction network of RetinaNet network is used for defect category and position prediction.

[0053] In one embodiment, the lightweight student network model in S200 includes a residual network with a depth of 18 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for defect result prediction; wherein the feature pyramid module first constructs multi-scale features with different scale sizes and the same number of channels using convolution layers, and then realizes top-down feature fusion using bilinear interpolation to realize the fusion of semantic features and pixel-level features, and the detection head module first encodes the fused multi-scale features using four convolution layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolution layers.

[0054] Specifically, the lightweight student network model is composed of a shallow neural network, including a residual network with a depth of 18 layers, a five-layer feature pyramid structure, and a prediction network based on RetinaNet. The residual network with a depth of 18 layers is used to extract features of defects of different scales, the five-layer feature pyramid is used for feature fusion between defects of different scales, and the prediction network of the RetinaNet network is used for defect class and position prediction.

[0055] S300: Training the teacher network model using the data set, and saving the trained teacher network model weight file.

[0056] Specifically, a defect detection teacher network model is constructed, trained, and saved. According to existing target detection methods, a detection network with large parameter quantity and strong performance is constructed as a teacher network model for defect detection, and the defect detection teacher network model is trained and tested through the labeled training set and test set. The model weight with excellent performance in the test set is saved.

[0057] S400: Loading the saved teacher network model weight file into the teacher network model, inputting the defect images in the data set into the teacher network model and the student network model to obtain first multi-scale features and second multi-scale features respectively, inputting the first multi-scale features and the second multi-scale features into the multi-expansion branch respectively to obtain scale perception enhanced first multi-scale features and scale perception enhanced second multi-scale features, inputting the first multi-scale features and the second multi-scale features into the fusion feature module respectively to obtain depth fused first multi-scale features and depth fused second multi-scale features, respectively passing through a plurality of expansion branches and feature fusion modules to obtain final depth fused first multi-scale features and final depth fused second multi-scale features, calculating a mixed multi-scale knowledge loss according to the final depth fused first multi-scale features and the final depth fused second multi-scale features, updating the parameters of the student network model using a back propagation algorithm based on a total loss composed of the mixed multi-scale knowledge loss and a prediction loss of the student network model, and obtaining a trained lightweight student network model.

[0058] Specifically, the training data is simultaneously input into the teacher network model and the lightweight student network model. On the one hand, by directly calculating the loss of defect classification prediction and defect position prediction, the accurate recognition of the network to the defect position and the class can be ensured. On the other hand, by calculating the loss between the mixed multi-scale knowledge of the teacher network model and the mixed multi-scale knowledge of the lightweight model, the knowledge learned by the teacher network model is transferred to the lightweight student network model to complete the knowledge migration. Combined with the classification and position prediction loss obtained by direct training and the mixed multi-scale knowledge loss obtained by knowledge distillation, the parameters of the network model are iteratively optimized to complete the training process and realize the accurate recognition of defects in the intelligent manufacturing process.

[0059] Further, the mixed multi-scale knowledge distillation method is mainly used to transfer the defect scale knowledge learned by the teacher network model with a complex model structure to the lightweight student network model, improve the cognitive ability of the lightweight student network model to defects, and further improve the detection performance.

[0060] As shown in FIG. 2, first, the multi-scale features extracted by the teacher network model and the lightweight student network model are input into the cascaded knowledge mixing module, and the corresponding mixed multi-scale knowledge is calculated. Among them, represents the mixed multi-scale knowledge of the teacher network model, represents the mixed multi-scale knowledge of the lightweight student network model. Then, by calculating the loss between the mixed multi-scale knowledge, the transfer of multi-scale mixed knowledge between the teacher network model and the lightweight student network model is realized. By transferring such mixed multi-scale knowledge, it is helpful to improve the cognitive ability of the lightweight defect detection network to defects of different scales, and further improve the recognition performance of defects.

[0061] Because the scale of the five-layer feature pyramid is fixed, which limits the representation ability of the network, resulting in that the scale knowledge of defects cannot be well distilled. By mixing knowledge of different scales, defect knowledge that integrates high-level semantics and low-level pixels can be obtained, which helps to improve the transfer of defect scale-related knowledge. The cascaded knowledge mixing module obtains deep mixed scale knowledge by cascading multiple multi-dilation convolution branches and feature mixing modules to improve the defect knowledge distillation process, thereby improving the recognition performance of the lightweight model to defects of different scales. The multi-dilation convolution branch is used to improve the representation ability of multi-scale features to defect scales. The feature fusion module is used to realize the deep mixing of knowledge between different scale features, and the integrated batch normalization is used to process the fused features, which can prevent data explosion and reduce the network training cost. The cascaded multi-dilation convolution branch and the feature fusion module can deeply fuse the multi-scale features, and the integrated batch normalization can optimize the network training process to extract deep mixed multi-scale knowledge.

[0062] In one embodiment, as shown in FIG. 3, the first multi-scale feature and the second multi-scale feature are input into the multi-dilation branch in S400 to obtain the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature, including:

[0063] S410: input the first multi-scale feature and the second multi-scale feature into the convolution layer branch with different dilation rates d, and then obtain the receptive field enhanced first multi-scale feature and the receptive field enhanced second multi-scale feature; wherein d = 1, 2, 3;

[0064] ​S420: splicing the receptive field enhanced first multi-scale feature and the first multi-scale feature, and splicing the receptive field enhanced second multi-scale feature and the second multi-scale feature to obtain new multi-scale features Or

[0065] S430: dimension reduction of the new multi-scale features by a 1x1 convolution layer to obtain a scale consistent with the original multi-scale feature dimension, to obtain the first multi-scale feature after perception enhancement and the second multi-scale feature after scale perception enhancement.

[0066] In one embodiment, the first multi-scale feature and the second multi-scale feature are respectively input into the multi-dilation branch in S400 to obtain the first multi-scale feature after scale perception enhancement and the second multi-scale feature after scale perception enhancement, specifically:

[0067] Wherein, i represents the feature of the ith scale, T represents the teacher detection model, L represents the lightweight defect detection model, conv di represents a 3x3 convolution with different dilation rates, C represents splicing along the channel dimension, represents the first multi-scale feature, represents the second multi-scale feature, conv1 represents a 1x1 convolution layer for dimension reduction, represents the first multi-scale feature after scale perception enhancement, represents the second multi-scale feature after scale perception enhancement.

[0068] Specifically, after the multi-dilation rate convolution branch, the network's perception ability for defect scale can be improved.

[0069] In one embodiment, as shown in FIG. 4, the first multi-scale feature after scale perception enhancement and the second multi-scale feature after scale perception enhancement are respectively input into the fusion feature module in S400 to obtain the first multi-scale feature after deep fusion and the second multi-scale feature after deep fusion, including:

[0070] S440: using bilinear interpolation technology and 3x3 convolution to interpolate the features adjacent to the first multi-scale feature after scale perception enhancement to obtain scale features consistent with the resolution size of the first multi-scale feature after scale perception enhancement; using bilinear interpolation technology and 3x3 convolution to interpolate the features adjacent to the second multi-scale feature after scale perception enhancement to obtain scale features consistent with the resolution size of the second multi-scale feature after scale perception enhancement;

[0071] S450: extracting, by using the deformable convolution, defect scale related features of the scale features with the same resolution size as the first multi-scale feature after the scale perception enhancement and the second multi-scale feature after the scale perception enhancement;

[0072] S460: adding the extracted defect scale related features respectively to realize fusion between adjacent scale features, to obtain the fused first multi-scale feature and the fused second multi-scale feature;

[0073] S470: processing the fused first multi-scale feature and the fused second multi-scale feature respectively by using the integrated normalization technology, and combining an activation function to ensure the nonlinearity of data, to obtain the first multi-scale feature after deep fusion and the second multi-scale feature after deep fusion.

[0074] Specifically, the feature fusion module is used for deep mixing of feature knowledge between different scales, so as to obtain multi-scale knowledge more adaptive to defect scales, and then to improve the knowledge distillation process.

[0075] In one embodiment, S440 is specifically:

[0076] wherein, F bi represents a bilinear interpolation technology, conv3 represents a 3x3 convolution layer with a step of 2, and represent adjacent scale features with the same resolution size as , and represent adjacent scale features with the same resolution size as ;

[0077] S450 is specifically:

[0078] wherein, dconv represents deformable convolution to extract defect scale related features, represents and defect scale related features extracted by deformable convolution, represents and defect scale related features extracted by deformable convolution.

[0079] In one embodiment, S460 is specifically:

[0080] wherein, and represent the first multi-scale feature after fusion and the second multi-scale feature after fusion;

[0081] S470 is specifically:

[0082] wherein, ReLU represents an activation function, IBN represents an integrated normalization function, and represent the first multi-scale feature after deep fusion and the second multi-scale feature after deep fusion, respectively.

[0083] Specifically, the use of deformable convolution further increases the field of view of the fixed scale feature, thereby further extracting knowledge more related to the defect size. The combination of deformable convolution and cross-scale fusion in S460 enables the resulting features to have stronger scale representation capabilities and contain more defect size-related knowledge.

[0084] To further deeply fuse defect size-related features, by cascading n (n ∈ N) multiple expansion rate convolutions and feature fusion modules, deep mixing of multi-scale features can be achieved and (wherein fn represents the number of cascades). Finally, the resulting and are processed by integrated normalization and activation functions to obtain the final mixed multi-scale knowledge, corresponding to the final deep fusion of the first multi-scale feature and the final deep fusion of the second multi-scale feature and to achieve mixed multi-scale feature extraction of the teacher network model and the lightweight student network model.

[0085] In one embodiment, the mixed multi-scale knowledge loss is calculated in S400 according to the final deep fusion of the first multi-scale feature and the final deep fusion of the second multi-scale feature, including:

[0086] wherein, Loss ms represents the mixed multi-scale knowledge loss, n represents the feature sample size, and represent the final deep fusion of the first multi-scale feature and the final deep fusion of the second multi-scale feature, respectively.

[0087] In one embodiment, the total loss in S400 is specifically: Loss = λ ms Loss ms +λ cls Loss cls +λ reg Lossreg

[0088] wherein λ ms , λ cls and λ reg adjust the balance between Loss ms , Loss cls and Loss reg respectively; Loss cls = -y(1-p) γ log(p)-(1-y)p γ log(1-p)

[0089] wherein y represents the real label of the defect category, p represents the predicted category, γ is a hyperparameter for adjusting the classification loss, and Loss cls is the classification prediction loss.

[0090] wherein IoU represents the intersection over union between the real label and the predicted frame, C represents the minimum closed shape, A c represents the area of C, U represents the sum of the areas of A and B, and Loss reg is the positioning loss.

[0091] Specifically, the prediction loss of the lightweight child network model mainly includes defect classification and position prediction loss, wherein the defect classification loss adopts the generalized focal loss Focal Loss, and the position prediction loss mainly adopts the GIoU Loss. By combining the real label of the dataset with the classification and position prediction results of the lightweight defect detection network, the corresponding classification prediction loss Loss cls and positioning loss Loss reg can be calculated.

[0092] The final total loss includes the prediction loss (Loss cls and Loss reg ) and the mixed multi-scale knowledge loss (Loss ms ). By combining the back propagation theory and the stochastic gradient descent algorithm, the parameters of the network model are updated, and through continuous iteration and optimization, the lightweight network can realize efficient and high-performance detection and recognition of defects in the intelligent manufacturing process under the constraint of loss.

[0093] S500: Deploy and apply the trained lightweight child network model to the autonomous mobile robot to realize dynamic and real-time defect detection of intelligent manufacturing products.

[0094] The above-mentioned lightweight defect detection method based on mixed multi-scale knowledge distillation transmits the knowledge learned by the teacher network model to the lightweight student network model to complete knowledge migration. The cascaded knowledge mixing module obtains deep mixed scale knowledge by cascading multiple multi-expansion rate convolution branches and feature mixing modules. By transmitting this mixed multi-scale knowledge, the defect knowledge distillation process is improved, which helps to improve the cognitive ability of the lightweight defect detection student network model to defects of different scales, and thus improves the defect recognition performance. Using a lightweight model for defect detection has the advantages of improving resource efficiency, enhancing real-time performance, and improving product quality, promoting the development of intelligent manufacturing, and realizing efficient, accurate, and sustainable intelligent manufacturing production.

[0095] The above describes in detail the lightweight defect detection method based on mixed multi-scale knowledge distillation provided by the present application. The principles and implementation modes of the present application are described using specific examples in this paper. The above examples are only used to help understand the core idea of the present application. It should be noted that those skilled in the art can make some improvements and modifications to the present application without departing from the principles of the present application. These improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A lightweight defect detection method based on hybrid multi-scale knowledge distillation, characterized in that, The method comprises the following steps: S100: photographing the intelligent manufacturing product, and performing a pretreatment operation on the photographed photo, using a marking software to mark, and constructing a data set for network training; S200: constructing a teacher network model and a lightweight student network model for intelligent manufacturing defect detection; S300: training the teacher network model using the data set, and saving the trained teacher network model weight file; S400: loading the saved teacher network model weight file to the teacher network model, inputting the defect images in the data set into the teacher network model and the student network model to obtain first multi-scale features and second multi-scale features respectively, inputting the first multi-scale features and the second multi-scale features into multiple inflation branches respectively to obtain scale perception enhanced first multi-scale features and scale perception enhanced second multi-scale features, inputting the scale perception enhanced first multi-scale features and the scale perception enhanced second multi-scale features into a fusion feature module respectively to obtain depth fused first multi-scale features and depth fused second multi-scale features, respectively passing through a plurality of inflation branches and a feature fusion module to obtain final depth fused first multi-scale features and final depth fused second multi-scale features, calculating a mixed multi-scale knowledge loss according to the final depth fused first multi-scale features and the final depth fused second multi-scale features, updating parameters of the student network model using a back propagation algorithm combined with a total loss composed of the mixed multi-scale knowledge loss and a prediction loss of the student network model, and obtaining a trained lightweight student network model; S500: deploying and applying the trained lightweight student network model to an autonomous mobile robot to realize dynamic and real-time defect detection of the intelligent manufacturing product.

2. The method of claim 1, wherein, The teacher network model in S200 comprises a residual network with a depth of 101 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for detection result prediction; wherein the feature pyramid module first constructs multi-scale features with different scale sizes and the same number of channels using a convolution layer, and then realizes top-down feature fusion using bilinear interpolation to realize fusion of semantic features and pixel-level features; the detection head module first encodes the fused multi-scale features using four convolution layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolution layers.

3. The method of claim 2, wherein, The lightweight student network model in S200 comprises a residual network with a depth of 18 layers for multi-scale feature extraction, a feature pyramid module for multi-scale feature fusion, and a detection head module for detection result prediction; wherein the feature pyramid module first constructs multi-scale features with different scale sizes and the same number of channels using a convolution layer, and then realizes top-down feature fusion using bilinear interpolation to realize fusion of semantic features and pixel-level features; the detection head module first encodes the fused multi-scale features using four convolution layers with shared weights, and then predicts the positioning box and the category of the defect through two decoding branches composed of convolution layers.

4. The method of claim 3, wherein, The first multi-scale feature and the second multi-scale feature are input into the multi-dilation branch in S400 to obtain the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature, including: S410: The first multi-scale feature and the second multi-scale feature are input into the convolution layer branch with different dilation rates d, and then the receptive field enhanced first multi-scale feature and the receptive field enhanced second multi-scale feature are obtained; wherein d=1, 2, 3; S420: splicing the first multi-scale feature with the receptive field enhanced first multi-scale feature and the second multi-scale feature with the receptive field enhanced second multi-scale feature to obtain new multi-scale features and S430: The new multi-scale feature is reduced in dimension by the 1x1 convolution layer to obtain the scale consistent with the dimension of the original multi-scale feature, and the perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature are obtained.

5. The method of claim 4, wherein, In S400, the first multi-scale feature and the second multi-scale feature are respectively input into the multi-expansion branch to obtain the scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature, specifically: wherein i represents the feature of the ith scale, T represents the teacher detection model, L represents the lightweight defect detection model, conv di represents a 3x3 convolution with different dilation rates, C represents concatenation along the channel dimension, representing a first multi-scale feature, denotes a second multi-scale feature, conv1 denotes a 1x1 convolutional layer for dimension reduction, representing the first multi-scale feature with a scale perception enhancement, The scale perception enhanced second multi-scale feature is represented.

6. The method of claim 5, wherein, The scale perception enhanced first multi-scale feature and the scale perception enhanced second multi-scale feature are input into the fusion feature module in S400 to obtain the depth fused first multi-scale feature and the depth fused second multi-scale feature, including: S440: The features adjacent to the scale perception enhanced first multi-scale feature are interpolated by the bilinear interpolation technology and the 3x3 convolution to obtain the scale feature consistent with the resolution size of the scale perception enhanced first multi-scale feature; the features adjacent to the scale perception enhanced second multi-scale feature are interpolated by the bilinear interpolation technology and the 3x3 convolution to obtain the scale feature consistent with the resolution size of the scale perception enhanced second multi-scale feature; S450: The deformable convolution is used to extract the defect scale related features of the scale perception enhanced first multi-scale feature and the scale feature consistent with the resolution size of the scale perception enhanced first multi-scale feature, and to extract the defect scale related features of the scale perception enhanced second multi-scale feature and the scale feature consistent with the resolution size of the scale perception enhanced first multi-scale feature; S460: The extracted defect scale related features are added respectively to realize the fusion between the adjacent scale features, and the fused first multi-scale feature and the fused second multi-scale feature are obtained; S470: The integrated normalization technology is used to process the fused first multi-scale feature and the fused second multi-scale feature respectively, and the activation function is combined to ensure the nonlinearity of the data, and the depth fused first multi-scale feature and the depth fused second multi-scale feature are obtained.

7. The method of claim 6, wherein, S440 is specifically: where F bi denotes bilinear interpolation technique, conv3 denotes a 3x3 convolution layer with a stride of 2, and Indicates having and adjacent scale features of the same resolution size, and Indicates having and Adjacent scale features with the same resolution size; S450 is specifically: wherein dconv denotes a deformable convolution to extract defect scale dependent features, indicates and defect scale related features extracted by deformable convolution, denotes and Defect scale related features extracted by the deformable convolution.

8. The method of claim 7, wherein, S460 is specifically: wherein, and The fused first multi-scale feature and the fused second multi-scale feature are represented. S470 is specifically: where ReLU denotes a rectified linear unit function, and IBN denotes an integrated batch normalization function, and The depth fused first multi-scale feature and the depth fused second multi-scale feature are represented respectively.

9. The method of claim 8, wherein, The mixed multi-scale knowledge loss is calculated in S400 according to the final depth-fused first multi-scale feature and the final depth-fused second multi-scale feature, including: wherein Loss ms denotes the mixed multi-scale knowledge loss, n denotes the feature sample size, and The final depth fused first multi-scale feature and the final depth fused second multi-scale feature are represented respectively.

10. The method of claim 9, wherein, The total loss in S400 is specifically: Loss = λ ms Loss ms + λ cls Loss cls + λ reg Loss reg where λ ms , λ cls and λ reg respectively adjust the balance between Loss ms , Loss cls and Loss reg losses; Loss cls = -y(1 - p) γ log(p) - (1 - y)p γ log(1 - p) where y represents the true label of the defect class, p represents the predicted class, and γ is a hyperparameter used to adjust the classification loss; where IoU denotes the intersection over union between the true label and the predicted box, C denotes the minimum enclosing shape, A c denotes the area of C, and U denotes the sum of the areas of A, B.

Images