Lightweight industrial glass defect detection method and system based on weighted mask distillation

Abstract

The invention discloses a lightweight industrial glass defect detection method and system based on weighted mask distillation, and relates to the technical field of digital image processing and target identification, a plurality of industrial glass images are acquired, defects are marked, and a data set is constructed; heterogeneous backbone feature extraction networks of the large-scale target detection model and the small-scale target detection model are built respectively, and pre-training is carried out by using a data set; guiding the small model backbone network to perform upstream knowledge distillation by taking the pre-trained large model backbone network as an upstream teacher model; a neck feature fusion network and a detection head which are isomorphic to the large-scale target detection model and the small-scale target detection model are built respectively and then connected with the corresponding backbone feature extraction network, and data sets are used for training respectively; taking the big model neck network obtained by training as a downstream teacher model, and guiding the small model neck network to perform downstream knowledge distillation; a lightweight small-scale target detection model is constructed, and the model is utilized to realize more efficient and more accurate defect detection of an industrial glass image to be detected.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing and target recognition, and in particular to a lightweight industrial glass defect detection method and system based on weighted mask distillation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Industrial quality inspection is crucial in the manufacturing industry, aiming to detect potential defects and non-conformities in products to ensure that they meet quality standards. Computer vision technology plays a key role in this field, using cameras to obtain images of the surface of industrial glass and using image processing algorithms to detect defects, inconsistencies and problems on the surface of industrial glass products, such as cracks, flaws, foreign matter and dimensional deviations, so as to achieve quality inspection for industrial glass.

[0004] In recent years, the field of computer vision technology has developed rapidly, involving image classification, object detection, and image segmentation. However, although using a larger network model can improve performance, it also requires more resources and reduces the inference speed, which is contrary to the needs of industrial applications. Therefore, these new models proposed in current research are difficult to be widely used in the field of industrial quality. Knowledge distillation is an effective solution that can transfer the knowledge of a large teacher network model to a small student network model, improving the performance of the student model without increasing resource requirements or reducing the inference speed. However, most knowledge distillation methods are designed for image classification. Although some feature-based distillation methods can be applied to various tasks, these general methods fail when directly applied to object detection. The main reasons are: (1) The traditional single distillation method imposes too strict constraints on the student model; (2) The characteristics of the teacher model and the student model are very different in different regions. Simply imitating the output of the teacher model will have a negative impact on the student model. For object detection, it is inefficient to simply let the student model simply imitate all the features of the teacher model, because the regions extracted during imitation that are not related to the target will introduce a lot of noise, which will affect the accuracy of the student model's object detection. Summary of the invention

[0005] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a lightweight industrial glass defect detection method and system based on weighted mask distillation, which adopts upstream and downstream knowledge distillation methods, selects reasonable distillation areas, and guides the training of lightweight small model glass defect target detection, thereby achieving more efficient and more accurate industrial glass defect detection, and effectively solving the problem of requiring lightweight and high accuracy at the same time in industrial defect detection scenarios.

[0006] In a first aspect, the present invention provides a lightweight industrial glass defect detection method based on weighted mask distillation.

[0007] A lightweight industrial glass defect detection method based on weighted mask distillation, comprising:

[0008] Obtain an image of industrial glass to be detected, input the image into a lightweight small-scale target detection model, and output the detected industrial glass defects; wherein the construction of the lightweight small-scale target detection model includes:

[0009] Obtain several industrial glass images and annotate defects to build a dataset;

[0010] Build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models respectively, and use data sets for pre-training respectively;

[0011] The pre-trained large model backbone network is used as the upstream teacher model to guide the small model backbone network to perform upstream knowledge distillation;

[0012] Build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, connect them to the corresponding backbone feature extraction networks, and train them separately using data sets;

[0013] The trained large model neck network is used as the downstream teacher model to guide the small model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation and global distillation.

[0014] Based on the network after upstream and downstream knowledge distillation, a lightweight small-scale target detection model is constructed.

[0015] A further technical solution is to construct the data set, including:

[0016] Acquire a number of industrial glass images, and mark the defect location and type of each industrial glass image;

[0017] Based on the annotated industrial glass images, a data set is constructed, and data enhancement is performed on the data set; the data enhancement includes unifying the image resolution and randomly flipping the image.

[0018] Further technical solutions are to build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models, including:

[0019] Build a backbone feature extraction network for large-scale object detection models, which consists of a cascade of single convolution blocks, separation convolution blocks, teacher feature extraction blocks, and downsampling blocks;

[0020] At the same time, a backbone feature extraction network of a small-scale target detection model is built, which is composed of a cascade of single convolution blocks, separation convolution blocks, student feature extraction blocks, and downsampling blocks;

[0021] Among them, the separable convolution block is composed of multiple single convolution blocks with different convolution kernels connected in series or in parallel. The teacher feature extraction block includes 5 single convolution blocks, 6 separable convolution blocks, residual edges, batch normalization, and Relu activation function. The student feature extraction block contains 4 single convolution blocks, 3 separable convolution blocks, residual edges, batch normalization, and Relu activation function. For the teacher feature extraction block and the student feature extraction block, the input image is first convolved with a single convolution block to reduce the number of channels, and then the features under different receptive fields are extracted by parallel separable convolution blocks. The extracted features are then batch normalized and Relu activated and then spliced ​​along the channel dimension for output.

[0022] A further technical solution is to use the pre-trained large model backbone network as the upstream teacher model to guide the small model backbone network to perform upstream knowledge distillation, including:

[0023] Input the images in the data set into the large and small model backbone networks respectively to obtain the corresponding output feature maps;

[0024] The output feature map of the large model backbone network and the output feature map of the small model backbone network are aligned in the spatial dimension through deconvolution, and then the upstream distillation loss function is used to calculate the loss of the aligned feature map; the loss function is:

[0025]

[0026] In the above formula, C, H, and W represent the channel, height, and width of the feature map respectively, and P represents the pixel point in the feature map;

[0027] After continuous iterative training, the loss is minimized, the upstream knowledge distillation is completed, and a trained small model backbone network is obtained.

[0028] A further technical solution is to build a neck feature fusion network and detection head that are isomorphic to the large-scale and small-scale target detection models, including: using the FPN feature pyramid network as the neck feature fusion network to output multi-scale feature maps; using RetinaHead as the detection head, and connecting the detection head to each output feature map of the neck feature fusion network;

[0029] Among them, the FPN feature pyramid network outputs different feature maps of multiple scales, which are used for the detection of large, medium and small targets respectively; RetinaHead includes two branches: classification and localization. The classification branch includes multiple intermediate convolutional layers connected in sequence and a classification output layer, which is used to predict the target category; the localization branch includes multiple intermediate convolutional layers connected in sequence and a bounding box regression layer, which is used to predict the target position;

[0030] The constructed neck feature fusion network and detection head are connected to the corresponding backbone feature extraction network, and the data sets are used for training respectively. During the training process, the L1 loss is used to calculate the model positioning loss, and the Focalloss is used to calculate the model classification loss.

[0031] Through continuous iterative training until the positioning and classification losses are minimized, the preliminary training of the neck feature fusion network and detection head of the large-scale and small-scale target detection models is completed.

[0032] A further technical solution is to use the trained large model neck network as the downstream teacher model to guide the small model neck network to perform downstream knowledge distillation, including:

[0033] The images in the dataset are input into the large and small models respectively, and the corresponding multi-scale feature maps are output through the neck network;

[0034] Align the feature maps of the large model and the small model of the same scale in the spatial dimension, and then use the binary mask and the scaling mask to separate the foreground and background of the feature map;

[0035] Based on the foreground feature maps of the large and small models, foreground distillation in downstream knowledge distillation is performed as follows: first, the foreground feature maps of the large and small models are feature activated through the hard attention module SimAm, and then masked according to the weighted score mask to obtain the masked foreground feature maps of the large and small models; then, under the guidance of the masked foreground feature map of the large model, the feature map of the masked foreground feature map of the small model is restored through the feature restoration module to obtain the foreground feature restoration map of the small model; according to the foreground feature restoration maps of the large and small models, the foreground loss and foreground attention loss are constructed;

[0036] Based on the background feature maps of large and small models, background distillation is performed in downstream knowledge distillation;

[0037] Based on the same-scale feature maps of the large and small models, global distillation is performed in the downstream knowledge distillation, which is: the feature maps of the large and small models fuse spatial information into the channel through the information fusion module to obtain the fused feature maps of the large and small models, so as to construct the global loss;

[0038] Taking the positioning and classification loss as the original loss, the foreground loss, background loss, foreground attention loss and global loss are combined to construct the total loss function. The dataset is used for continuous iterative training until the total loss is minimized, thus completing the training of the small-scale target detection model.

[0039] A further technical solution is that the calculation process of the weighted score mask is:

[0040] Fill and split the feature map into A feature block of size α, where H and W represent the height and width of the feature map;

[0041] The feature block size is converted to And calculate the feature block weight;

[0042] Generate a value less than The random number R is used to index the corresponding feature block and add the feature block to the storage block M;

[0043] Calculate the feature blocks whose distance from any feature block in the storage block M is less than β among all feature blocks, multiply the obtained feature block by the weight of its corresponding feature block to obtain a weighted score, select the feature block with the highest score, add it to the storage block M, and repeat the above operation until the number of repetitions reaches the set number;

[0044] Generate a matrix of size (1, H, W), and according to the feature blocks in the obtained storage block M, set the corresponding part of the matrix to 0 and the rest to 1 to obtain the weighted score mask.

[0045] In a second aspect, the present invention provides a lightweight industrial glass defect detection system based on weighted mask distillation.

[0046] A lightweight industrial glass defect detection system based on weighted mask distillation, comprising:

[0047] Glass image acquisition module, used to acquire the image of industrial glass to be tested.

[0048] The glass defect detection module is used to input the image into the lightweight small-scale target detection model and output the detected industrial glass defects. The construction of the lightweight small-scale target detection model includes the following modules:

[0049] The dataset construction module is used to obtain several industrial glass images and annotate defects to construct a dataset;

[0050] The upstream knowledge distillation module is used to build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models, and pre-train them using data sets. The pre-trained backbone network of the large model is used as the upstream teacher model to guide the backbone network of the small model to perform upstream knowledge distillation.

[0051] The downstream knowledge distillation module is used to build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, respectively, and then connect them with the corresponding backbone feature extraction networks and train them separately using data sets; the trained large-model neck network is used as the downstream teacher model to guide the small-model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation, and global distillation;

[0052] The small model building module is used to build a lightweight small-scale target detection model based on the network after upstream and downstream knowledge distillation.

[0053] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned lightweight industrial glass defect detection method based on weighted mask distillation when executing the executable instructions stored in the memory.

[0054] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned lightweight industrial glass defect detection method based on weighted mask distillation.

[0055] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned lightweight industrial glass defect detection method based on weighted mask distillation is implemented.

[0056] One or more of the above technical solutions have the following beneficial effects:

[0057] 1. The present invention provides a lightweight industrial glass defect detection method and system based on weighted mask distillation. It adopts upstream and downstream knowledge distillation, selects reasonable distillation areas, guides the training of lightweight small model glass defect target detection, and achieves more efficient and more accurate industrial glass defect detection, effectively solving the problem of requiring lightweight and high accuracy in industrial defect detection scenarios.

[0058] 2. The present invention proposes an upstream + downstream knowledge distillation method. Through upstream knowledge distillation, the features of the model pre-training stage (i.e., task-independent generalized image feature knowledge) are distilled, and then through downstream knowledge distillation, the task-related feature knowledge learned for specific tasks is distilled. Through upstream and downstream knowledge distillation, the decoupled distillation of task-independent knowledge and task-related knowledge is achieved, thereby ensuring the accuracy of the final distillation learning.

[0059] 3. In the downstream knowledge distillation process, the present invention introduces foreground and background separation distillation and global distillation. Through foreground and background separation, the student model can focus on the foreground area and concentrate on learning the key features related to defects in the teacher model without being disturbed by irrelevant background information. Through global distillation, more global context information can be integrated into the channel to solve the problem of boundary information loss in the distillation process. Among them, in order to solve the problem that simply making the student model imitate the features of the teacher model in the knowledge distillation process will impose overly strict constraints on the student features, making the knowledge distillation relatively inefficient, the present invention also proposes a weighted score mask + mask feature restoration method, which changes the target task in the distillation process from feature imitation to feature restoration. This weakly constrained distillation method can enable the student model to learn more advanced features of the teacher model and ensure the effectiveness of the final knowledge distillation.

[0060] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0062] Figure 1 A flowchart of constructing a lightweight small-scale target detection model in an embodiment of the present invention;

[0063] Figure 2 It is a structural diagram of the backbone feature extraction network of the large-scale target detection model in an embodiment of the present invention;

[0064] Figure 3 It is a structural diagram of the backbone feature extraction network of the small-scale target detection model in an embodiment of the present invention;

[0065] Figure 4 It is a structural diagram of a teacher feature extraction block in an embodiment of the present invention;

[0066] Figure 5 It is a structural diagram of a student feature extraction block in an embodiment of the present invention;

[0067] Figure 6 A schematic diagram of downstream knowledge distillation in an embodiment of the present invention;

[0068] Figure 7 This is a structural diagram of foreground knowledge distillation in downstream knowledge distillation in an embodiment of the present invention;

[0069] Figure 8 This is a diagram of the global knowledge distillation structure in the downstream knowledge distillation in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0071] Embodiment 1

[0072] This embodiment provides a lightweight industrial glass defect detection method based on weighted mask distillation, the method comprising:

[0073] Obtain the industrial glass image to be detected, input the image into the lightweight small-scale target detection model, and output the detected industrial glass defects.

[0074] The main purpose of this embodiment is to build a lightweight small-scale target detection model (referred to as a small model). The construction process of the small model is as follows: Figure 1 As shown, the specific steps include:

[0075] Step S1: Acquire several industrial glass images and annotate defects to build a data set.

[0076] Specifically, a gray-point industrial camera is used to shoot industrial glass images or pictures. After obtaining several industrial glass images, the defect locations and types in each industrial glass picture are manually annotated using annotation tools. For the annotated industrial glass images, an annotation file in COCO format is exported to complete the construction of the data set (i.e., COCO data set). The annotation file contains information such as image width, height, label coordinates, and classification. The coordinate format is "xywh", that is, there are four variables in the coordinates. The x and y variables represent the horizontal and vertical coordinates of the upper left corner of the target frame, respectively, and w and h represent the width and height of the target frame, respectively.

[0077] Furthermore, after the dataset is constructed, before the dataset is loaded into the model for training, data enhancement is performed on the dataset, including: adjusting the resolution of all images to 480×480 and randomly flipping the images, with the flipping probability being a set value of 0.5.

[0078] Step S2: Build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models respectively, and use the data sets for pre-training respectively.

[0079] First, we build the backbone feature extraction networks of large-scale and small-scale target detection models (referred to as large and small models), respectively. Figure 2 As shown in , the backbone feature extraction network of the large model is composed of a cascade of single convolution blocks, separation convolution blocks, downsampling blocks, and teacher feature extraction blocks; Figure 3 As shown, the small-scale target detection network is composed of a single convolution block, a separation convolution block, a downsampling block, and a student feature extraction block cascaded. Preferably, in order to ensure a certain degree of isomorphism and similarity between the large and small models on the basis of a lightweight small model, this embodiment also introduces a small number of teacher feature extraction blocks when designing the small model backbone extraction network, that is, compared with 12 teacher feature extraction blocks in the large model backbone extraction network, only 2 teacher feature extraction blocks are set in the small model backbone extraction network, and then 4 downsampling blocks and student feature extraction blocks are set.

[0080] Among them, the single convolution block is divided into a 1x1 single convolution block and a 3x3 single convolution block; the separation convolution block is composed of a 1x3 convolution block and a 3x1 convolution block (i.e., convolution blocks with different convolution kernels) connected in series or in parallel; Figure 4 As shown in , the teacher feature extraction block includes 5 single convolution blocks, 6 separation convolution blocks, residual edges, batch normalization, and ReLU activation function; Figure 5 As shown in the figure, the student feature extraction block is more lightweight than the teacher feature extraction block. It contains 4 single convolution blocks, 3 separation convolution blocks, residual edges, batch normalization, and ReLU activation function.

[0081] Based on the above settings, the large model can obtain more advanced image features by setting up more convolution modules and more complex feature extraction block cascades, which can make the detection results output by the large model more accurate and provide more effective information for subsequent distillation; while the small model uses relatively fewer convolution modules and simpler feature extraction block cascades to ensure detection speed. At the same time, this design can achieve structural similarity between the large model and the small model, facilitate the alignment of model features during subsequent knowledge distillation, and improve knowledge transfer efficiency.

[0082] Furthermore, based on the above network structure, the teacher feature extraction block and the student feature extraction block first convolve the input image through a 1x1 single convolution block to reduce the number of channels; then extract features under different receptive fields through parallel separation convolution blocks; and then the extracted features are batch normalized and Rel u After the activation function, the blocks are concatenated along the channel dimension and output. The feature extraction formulas of the teacher feature extraction block and the student feature extraction block are as follows:

[0083]

[0084] Among them, F′ S is the output feature map of the student feature extraction block, F′T is the output feature map of the teacher feature extraction block, It is an operation that changes the number of feature map channels.

[0085] In the above formula, the branch formulas of feature extraction are as follows:

[0086] Branch1(F)=Relu(BN(Conv(F));

[0087] Branch2(F)=Relu(BN(SeConv(Conv(F)));

[0088]

[0089] Among them, Branch1, Branch2, Branch S , Represents different branches of the feature extraction block, Conv represents a single convolution block with a convolution kernel size of 1x1, SeConv is a serial separation convolution block, PaConv is a parallel separation convolution block, F is the input feature map of the feature extraction block, Relu is the activation function, and BN represents batch normalization.

[0090] Secondly, after completing the construction of the backbone feature extraction network, first add a fully connected layer to the last layer of the backbone feature extraction network of the large model and the small model respectively, and perform pre-training for a set number of times, such as 80 rounds, on the dataset constructed in the above step S1 (i.e., the ImageNet dataset). After the pre-training is completed, remove the fully connected layer.

[0091] Step S3: Use the pre-trained large model backbone network as the upstream teacher model to guide the small model backbone network to perform upstream knowledge distillation.

[0092] Considering that knowledge distillation in traditional target detection is usually only downstream knowledge distillation, the downstream knowledge refers to the composite feature knowledge obtained by first obtaining task-independent generalized image feature knowledge through general pre-training, and then conducting targeted training on a specific data set for a specific task. Different from the traditional method of only performing downstream knowledge distillation to obtain composite feature knowledge, this embodiment also introduces upstream knowledge distillation, through which the features of the model pre-training stage (i.e., task-independent generalized image feature knowledge) are distilled, and then the task-related feature knowledge obtained from learning for a specific task is distilled through downstream knowledge distillation. The decoupled distillation of task-independent knowledge and task-related knowledge is achieved through upstream and downstream knowledge distillation, thereby ensuring the accuracy of the final distillation learning.

[0093] Specifically, the images in the ImageNet dataset are input into the large and small model backbone networks respectively, and the output feature map of the large model backbone network and the output feature map of the small model backbone network are obtained (the output feature map here is the feature map finally output by the backbone network); the output feature map of the large model backbone network and the output feature map of the small model backbone network are aligned in size in the spatial dimension through deconvolution; and then the upstream distillation loss function is used to calculate the loss of the aligned feature map, and the loss function is:

[0094]

[0095] Among them, C, H, and W represent the channel, height, and width of the feature map respectively, and P represents the pixel point in the feature map.

[0096] The dataset is used to perform continuous iterative training until the loss is minimized, the upstream knowledge distillation is completed, and the trained large and small model backbone networks are obtained.

[0097] Step S4: Build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, respectively, connect them with the corresponding backbone feature extraction networks, and use the data sets for training respectively.

[0098] Specifically, the large-scale target detection model and the small-scale target detection model use the same neck feature fusion network and detection head, where (1) the FPN feature pyramid network is used as the neck feature fusion network. The FPN feature pyramid network can output different feature maps of multiple scales. In this embodiment, it is set to output five feature maps of different scales, three of which are obtained by feature fusion through four output feature maps of the backbone network (the output feature maps here are multi-scale feature maps extracted by multiple feature extraction blocks in the backbone network respectively, and in this embodiment, they are four feature maps output by the last layer of feature extraction blocks and the three layers of feature extraction blocks above them) for medium and small target detection; two of them are obtained by a 3×3 convolution on the last layer output of the backbone network for large target detection; (2) RetinaHead is used as the detection head of the target detection model, connected to each output feature map of the neck feature fusion network, where RetinaHead includes two branches, classification and positioning. The classification branch is used to predict the target category, which includes four 3×3 intermediate convolution layers and a classification output layer; the positioning branch is used to predict the target position, which includes four 3×3 intermediate convolution layers and a bounding box regression layer.

[0099] Based on the above network structure, the target detection loss function is constructed, including: using L1 loss to calculate the model positioning loss, and using Focalloss to calculate the model classification loss. The calculation formula of Focalloss is:

[0100] Focalloss(p)=-a(1-p) λ log(p);

[0101] Among them, a is the category weight, which is used to balance the imbalance of positive and negative samples; λ represents the weight of difficult samples, which is used to weigh difficult samples and easy samples.

[0102] Furthermore, the constructed neck feature fusion network and detection head are connected to the corresponding backbone feature extraction network, and trained respectively using the data sets, including: first, pre-training is performed using the COCO data set to obtain pre-training weights, and the pre-training weights loaded on the COCO data set are loaded when the neck feature fusion network of the large-scale target detection model and the small-scale target detection model are initialized, and the pre-training weights loaded on the COCO data set are loaded for the intermediate convolutional layers of the two branches of RetinaHead. The classification output layer and the bounding box regression layer are initialized with random values ​​between 0 and 1, and the classification output layer is set The number of classifications is 4; secondly, the processed data set in step S1 is used to train the large model and the small model using the SGD optimizer for 300 rounds, with an initial learning rate of 0.01; the learning rate is used for warm-up for 5 rounds, and the final learning rate increases to 0.1; the learning rate decay strategy is used to reduce the learning rate by 10 times in the 80th, 160th, and 240th rounds; the gradient clipping strategy is used, the norm type is specified as the L2 norm, and the maximum threshold is 30; after continuous iterative training, the positioning and classification losses are minimized, and the preliminary training of the neck feature fusion network and the detection head of the large and small-scale target detection models is completed.

[0103] Step S5: Use the trained large model neck network as the downstream teacher model to guide the small model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation and global distillation.

[0104] Step S5.1, input the images in the data set into the large and small models respectively, and output the corresponding multi-scale feature maps through the neck network. In this embodiment, five feature maps of different sizes output from the large model neck feature fusion network and the small model neck feature fusion network are obtained respectively.

[0105] Step S5.2: Align the size of the same-scale feature maps obtained by the large model and the small model in the spatial dimension through deconvolution; change the format of the real box label from "xywh" to "xyxy", and adjust the label value accordingly according to the mask filling and scaling ratio of the original image in the data enhancement stage; then, use the binary mask and scaling mask to separate the foreground and background of the feature map, including:

[0106] (1) Set a binary mask to separate foreground and background information. Set the target inside the real frame to 1, i.e., foreground information, and set the target outside the real frame to 0, i.e., background information, which can be expressed as:

[0107]

[0108] Where x and y are the horizontal and vertical coordinates of the pixel, and GT is the real frame area. When the position of x and y falls within the real frame area, FBMask is set to 1, otherwise it is set to 0.

[0109] (2) At the same time, the scaling mask technology is used to solve the imbalance of sample distribution. The formula is:

[0110]

[0111] Among them, H GT and W GT Represents the height and width of the real box respectively. When a pixel is in the bounding box of multiple targets at the same time, the bounding box of the smaller object is selected to calculate its ScaleMask.

[0112] Furthermore, considering that the background area in the industrial defect detection environment is complex and often contains a large amount of irrelevant or interfering information, which may have an adverse effect on the knowledge distillation effect, this embodiment introduces a foreground and background separation distillation method. Through foreground and background separation, the student model can focus on the foreground area and concentrate on learning the key features related to defects in the teacher model without being disturbed by irrelevant background information; further, considering that there is a certain semantic similarity between the boundary of the foreground area and the background area in the industrial defect dataset, that is, the background points close to the foreground area also contain some semantic information of the defect features, and simple foreground and background separation distillation will directly classify this part of the semantic information as non-defect sample information. For this reason, this embodiment also introduces global distillation, so that more global context information can be integrated into the channel to solve the problem of boundary information loss during the distillation process. That is, Figure 6 As shown, this embodiment performs foreground distillation, background distillation and global distillation respectively.

[0113] Step S5.3, based on the foreground feature maps of the large and small models, perform foreground distillation in downstream knowledge distillation. Specifically, considering that simply making the student model imitate the features of the teacher model in the knowledge distillation process will impose overly strict constraints on the student features, making knowledge distillation relatively inefficient, this embodiment proposes the above-mentioned weighted score mask + mask feature restoration method, which changes the target task in the distillation process from feature imitation to feature restoration, which superficially increases the difficulty of the distillation task, but this weakly constrained distillation method allows the student model to learn more advanced features of the teacher model.

[0114] like Figure 7As shown in the figure, for the foreground distillation part in the downstream knowledge distillation, the foreground feature maps of the large and small models are first activated by a hard attention module SimAm, and then masked according to the weighted score mask to obtain the masked foreground feature maps of the large and small models; then, under the guidance of the masked foreground feature map of the large model, the masked foreground feature map of the small model is restored through a feature restoration module to obtain the small model foreground recovery feature map and the large model foreground recovery feature map (i.e., the large model foreground feature map).

[0115] Furthermore, the calculation process of the weighted score mask is as follows:

[0116] First, pad the feature map and cut it into A feature block of size α, where H and W represent the height and width of the feature map; then, the feature map is transformed into a size of Calculate the feature block weight map (or feature block weight); secondly, generate a smaller than The random number R is used to index the corresponding feature block, and the feature block is added to the storage block M. The storage block M is a storage area, which only exists during the mask generation process. The feature blocks contained in it gradually increase with the mask calculation and are destroyed when the mask generation is completed; calculate the feature blocks whose distance from any feature block in the storage block M is less than β among all feature blocks; then, multiply the obtained feature block with its corresponding block weight to obtain its weighted score, select the feature block with the highest score, add it to the storage block M, and repeat the above operation until the number of repetitions reaches times. Among them, ω represents the masking rate, and the calculation formula of the weight score can be expressed as:

[0117]

[0118] Where x, y represent the height and width index of the feature block respectively; M represents the storage block used to generate the mask; D x,y It represents the minimum Euclidean distance between x, y and any feature block in M; δ is a hyperparameter, ε is Gaussian noise with mean 0 and standard deviation δ; S represents the weight score of the feature block.

[0119] Subsequently, a matrix of size (1, H, W) is generated. According to the feature blocks in the storage block M obtained in the previous step, the corresponding parts in the matrix are set to 0 and the rest to 1, and the weighted score mask can be obtained.

[0120] Compared with random mask, the weighted score mask proposed in this embodiment is a mask generation method that is closer to uniform distribution. The weighted score mask selects the appropriate masking area through comprehensive consideration of weight score and feature block distance, making the mask more reasonable and facilitating subsequent precise knowledge distillation.

[0121] Furthermore, the hard attention module SimAm above determines the importance of each neuron by calculating the energy function, which can be expressed as:

[0122]

[0123]

[0124] Among them, t represents the target neuron, H and W are the height and width of the feature map, H×W represents the number of neurons on the channel, and x i represents the non-target neurons on this channel, represents the final energy function.

[0125] The final generated foreground attention map can be expressed as:

[0126]

[0127] Among them, E means all Grouping on channel and spatial dimensions.

[0128] Through the above-mentioned hard attention module, the points with high response degree in the feature map are further activated, focusing on the key areas of distillation, so as to better realize feature restoration in the future.

[0129] In the above feature restoration module, a convolution module with a convolution kernel size of 3×3, a convolution module with a convolution kernel size of 5×5, and a convolution module with a convolution kernel size of 7×7 are used to restore features under different receptive fields, and a relu activation function is added therein; after the obtained feature map is restored to the original feature map size, it is superimposed on the channel dimension; finally, a convolution module with a convolution kernel size of 1×1 is used to fuse the information of the channel dimension to obtain the restored feature map. The formula of the feature restoration module is:

[0130]

[0131] Among them, F m represents the feature map after being masked by the weighted score mask, F p The restored feature map; Conv1, Conv2, and Conv3 represent convolution modules with kernel sizes of 3×3, 5×5, and 7×7, respectively, which are used for feature restoration; Conv4 represents a convolution module with a kernel size of 1×1, which is used for feature fusion.

[0132] Step S5.4: Based on the background feature maps of the large and small models, background distillation in downstream knowledge distillation is performed.

[0133] Specifically, for the background distillation part in the downstream knowledge distillation, the L2 loss of the background feature map is directly calculated, and a hyperparameter is added to achieve loss balance.

[0134] Step S5.5: Based on the same-scale feature maps of the large and small models, global distillation is performed in downstream knowledge distillation.

[0135] Specifically, Figure 8 As shown in the figure, for the global distillation part in the downstream knowledge distillation, the feature maps of the large and small models are first fused into the channel through an information fusion module to obtain the fused feature maps of the large and small models, and then the L2 loss of the two fused feature maps is calculated, which is the global loss.

[0136] In the information fusion module, the feature map first passes through a convolution module with a convolution kernel size of 7×7 for spatial information fusion; at the same time, the feature map passes through a convolution module with a convolution kernel size of 1×1 and performs softmax to obtain the attention matrix; the attention matrix is ​​multiplied with the feature map after spatial information fusion, and the multiplied feature map is first reduced in dimension through a 1×1 convolution to compress the information into the channel, and then passes through batch normalization and ReLU activation function; finally, the feature map is restored to the original dimension through a 1×1 convolution. The formula of the information fusion module is:

[0137]

[0138] Among them, Conv3 represents a 7×7 convolutional layer, Conv1, Conv2, and Conv4 represent 1×1 convolutional layers, ReLU is the activation function, a and b represent pixels in the feature map, σ represents variance, μ represents mean, and γ and β represent learnable parameters in normalization.

[0139] Step S5.6: Take the positioning and classification loss as the original loss, combine the foreground loss, background loss, foreground attention loss and the global loss based on the multi-scale feature map, construct the total loss function, use the data set to perform continuous iterative training until the total loss is minimized, and complete the training of the small-scale target detection model.

[0140] Specifically, the total loss function in the downstream knowledge distillation process consists of five parts: foreground loss, background loss, foreground attention loss, global loss, and original loss of the student model. The foreground and background loss formulas are:

[0141]

[0142]

[0143] Among them, L fg is the foreground loss, L bgis the background loss, FBMask represents the foreground-background separation mask, ScaleMask represents the scaling mask, AttentionMask represents the SimAm attention mask, and F T and F S They represent the feature maps of the teacher model (i.e., the large model) and the student model (i.e., the small model), respectively. C, H, and W correspond to the channel, height, and width of the feature map, respectively. α and β are hyperparameters for balancing the loss. R represents mask feature restoration, f represents feature alignment operation, and S is a random mask.

[0144] The above global distillation loss is constructed based on the multi-scale feature map, and the loss is:

[0145]

[0146] Among them, M represents the number of FPN layers, l corresponds to the FPN layer of the first layer, σ represents the information fusion module, ρ represents the feature map alignment operation, and F T and F S denote the feature maps of the teacher model and the student model respectively, and γ is a hyperparameter used to balance the loss.

[0147] The foreground attention loss is a loss reserved for the traditional distillation method with feature imitation as the distillation target. By adjusting the hyperparameters used to balance the loss, the foreground loss can be supplemented. The foreground attention loss is:

[0148]

[0149] Among them, C, H, and W correspond to the channel, height, and width of the feature map respectively; F T and F S denote the feature maps of the teacher model and the student model respectively, and δ is a hyperparameter for balancing the loss.

[0150] The overall loss of the final distillation process is as follows:

[0151] L Distill =L fg +L bg +L Global +L Attention +L Student ;

[0152] Among them, L Student It is the original loss generated by the student model in RetinaHead, that is, the target detection loss function of the small model in the above step S4, specifically refers to the loss caused by the difference between the prediction result of the small model and the true value.

[0153] Step S6: Based on the network after upstream and downstream knowledge distillation, a lightweight small-scale target detection model is constructed.

[0154] Specifically, through the above-mentioned upstream and downstream knowledge distillation and training, a lightweight small-scale target detection model is constructed based on the backbone feature extraction network, neck feature fusion network and detection head of the trained small model. The model is encapsulated in ONNX format, and then the model reasoning is accelerated through the TensorRT acceleration library.

[0155] Embodiment 2

[0156] This embodiment provides a lightweight industrial glass defect detection system based on weighted mask distillation, including:

[0157] A glass image acquisition module, used to acquire the image of industrial glass to be inspected;

[0158] The glass defect detection module is used to input the image into the lightweight small-scale target detection model and output the detected industrial glass defects. The construction of the lightweight small-scale target detection model includes the following modules:

[0159] The dataset construction module is used to obtain several industrial glass images and annotate defects to construct a dataset;

[0160] The upstream knowledge distillation module is used to build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models, and pre-train them using data sets. The pre-trained backbone network of the large model is used as the upstream teacher model to guide the backbone network of the small model to perform upstream knowledge distillation.

[0161] The downstream knowledge distillation module is used to build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, respectively, and then connect them with the corresponding backbone feature extraction networks and train them separately using data sets; the trained large-model neck network is used as the downstream teacher model to guide the small-model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation, and global distillation;

[0162] The small model building module is used to build a lightweight small-scale target detection model based on the network after upstream and downstream knowledge distillation.

[0163] Embodiment 3

[0164] This embodiment provides an electronic device, including: a memory, used to store executable instructions; and a processor, used to implement the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0165] Embodiment 4

[0166] This embodiment also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0167] Embodiment 5

[0168] This embodiment provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the above method provided in this embodiment.

[0169] The steps involved in the above embodiments 2 to 5 correspond to the method embodiment 1. For the specific implementation, please refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0170] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0171] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A lightweight industrial glass defect detection method based on weighted mask distillation, characterized in that: include: Obtain an image of industrial glass to be detected, input the image into a lightweight small-scale target detection model, and output the detected industrial glass defects; wherein the construction of the lightweight small-scale target detection model includes: Obtain several industrial glass images and annotate defects to build a dataset; Build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models respectively, and use data sets for pre-training respectively; The pre-trained large model backbone network is used as the upstream teacher model to guide the small model backbone network to perform upstream knowledge distillation; Build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, connect them to the corresponding backbone feature extraction networks, and train them separately using data sets; The trained large model neck network is used as the downstream teacher model to guide the small model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation and global distillation. Based on the network after upstream and downstream knowledge distillation, a lightweight small-scale target detection model is constructed.

2. A lightweight industrial glass defect detection method based on weighted mask distillation as claimed in claim 1, characterized in that: Build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models, including: Build a backbone feature extraction network for large-scale object detection models, which consists of a cascade of single convolution blocks, separation convolution blocks, teacher feature extraction blocks, and downsampling blocks; At the same time, a backbone feature extraction network of a small-scale target detection model is built, which is composed of a cascade of single convolution blocks, separation convolution blocks, student feature extraction blocks, and downsampling blocks; Among them, the separable convolution block is composed of multiple single convolution blocks with different convolution kernels connected in series or in parallel. The teacher feature extraction block includes 5 single convolution blocks, 6 separable convolution blocks, residual edges, batch normalization, and Relu activation function. The student feature extraction block contains 4 single convolution blocks, 3 separable convolution blocks, residual edges, batch normalization, and Relu activation function. For the teacher feature extraction block and the student feature extraction block, the input image is first convolved with a single convolution block to reduce the number of channels, and then the features under different receptive fields are extracted by parallel separable convolution blocks. The extracted features are then batch normalized and Relu activated and then spliced ​​along the channel dimension for output.

3. The lightweight industrial glass defect detection method based on weighted mask distillation according to claim 1, characterized in that: The pre-trained large model backbone network is used as the upstream teacher model to guide the small model backbone network to perform upstream knowledge distillation, including: Input the images in the data set into the large and small model backbone networks respectively to obtain the corresponding output feature maps; The output feature map of the large model backbone network and the output feature map of the small model backbone network are aligned in the spatial dimension through deconvolution, and then the upstream distillation loss function is used to calculate the loss of the aligned feature map; the loss function is: In the above formula, C, H, and W represent the channel, height, and width of the feature map respectively, and P represents the pixel point in the feature map; After continuous iterative training, the loss is minimized, the upstream knowledge distillation is completed, and a trained small model backbone network is obtained.

4. The lightweight industrial glass defect detection method based on weighted mask distillation according to claim 1, characterized in that: The neck feature fusion network and detection head with the same structure as the large-scale and small-scale target detection models are respectively constructed, including: using the FPN feature pyramid network as the neck feature fusion network to output multi-scale feature maps; using RetinaHead as the detection head, and connecting the detection head to each output feature map of the neck feature fusion network; Among them, the FPN feature pyramid network outputs different feature maps of multiple scales, which are used for the detection of large, medium and small targets respectively; RetinaHead includes two branches: classification and localization. The classification branch includes multiple intermediate convolutional layers connected in sequence and a classification output layer, which is used to predict the target category; the localization branch includes multiple intermediate convolutional layers connected in sequence and a bounding box regression layer, which is used to predict the target position; The constructed neck feature fusion network and detection head are connected to the corresponding backbone feature extraction network, and the data sets are used for training respectively. During the training process, the L1 loss is used to calculate the model positioning loss, and the Focalloss is used to calculate the model classification loss. Through continuous iterative training until the positioning and classification losses are minimized, the preliminary training of the neck feature fusion network and detection head of the large-scale and small-scale target detection models is completed.

5. The lightweight industrial glass defect detection method based on weighted mask distillation according to claim 1, characterized in that: The trained large model neck network is used as the downstream teacher model to guide the small model neck network to perform downstream knowledge distillation, including: The images in the dataset are input into the large and small models respectively, and the corresponding multi-scale feature maps are output through the neck network; Align the feature maps of the large model and the small model of the same scale in the spatial dimension, and then use the binary mask and the scaling mask to separate the foreground and background of the feature map; Based on the foreground feature maps of the large and small models, foreground distillation in downstream knowledge distillation is performed as follows: first, the foreground feature maps of the large and small models are feature activated through the hard attention module SimAm, and then masked according to the weighted score mask to obtain the masked foreground feature maps of the large and small models; then, under the guidance of the masked foreground feature map of the large model, the feature map of the masked foreground feature map of the small model is restored through the feature restoration module to obtain the foreground feature restoration map of the small model; according to the foreground feature restoration maps of the large and small models, the foreground loss and foreground attention loss are constructed; Based on the background feature maps of large and small models, background distillation is performed in downstream knowledge distillation; Based on the same-scale feature maps of the large and small models, global distillation is performed in the downstream knowledge distillation, which is: the feature maps of the large and small models fuse spatial information into the channel through the information fusion module to obtain the fused feature maps of the large and small models, so as to construct the global loss; Taking the positioning and classification loss as the original loss, the foreground loss, background loss, foreground attention loss and global loss are combined to construct the total loss function. The dataset is used for continuous iterative training until the total loss is minimized, thus completing the training of the small-scale target detection model.

6. A lightweight industrial glass defect detection method based on weighted mask distillation as claimed in claim 5, characterized in that: The calculation process of the weighted score mask is: Fill and split the feature map into A feature block of size α, where H and W represent the height and width of the feature map; The feature block size is converted to And calculate the feature block weight; Generate a value less than The random number R is used to index the corresponding feature block and add the feature block to the storage block M; Calculate the feature blocks whose distance from any feature block in the storage block M is less than β among all feature blocks, multiply the obtained feature block by the weight of its corresponding feature block to obtain a weighted score, select the feature block with the highest score, add it to the storage block M, and repeat the above operation until the number of repetitions reaches the set number; Generate a matrix of size (1, H, W), and according to the feature blocks in the obtained storage block M, set the corresponding parts in the matrix to 0 and the rest to 1 to obtain the weighted score mask.

7. A lightweight industrial glass defect detection system based on weighted mask distillation, characterized in that: include: Glass image acquisition module, used to acquire the image of industrial glass to be tested. The glass defect detection module is used to input the image into the lightweight small-scale target detection model and output the detected industrial glass defects. The construction of the lightweight small-scale target detection model includes the following modules: The dataset construction module is used to obtain several industrial glass images and annotate defects to construct a dataset; The upstream knowledge distillation module is used to build heterogeneous backbone feature extraction networks for large-scale and small-scale target detection models, and pre-train them using data sets. The pre-trained backbone network of the large model is used as the upstream teacher model to guide the backbone network of the small model to perform upstream knowledge distillation. The downstream knowledge distillation module is used to build neck feature fusion networks and detection heads that are isomorphic to large-scale and small-scale target detection models, respectively, and then connect them with the corresponding backbone feature extraction networks and train them separately using data sets; the trained large-model neck network is used as the downstream teacher model to guide the small-model neck network to perform downstream knowledge distillation, including foreground distillation, background distillation, and global distillation; The small model building module is used to build a lightweight small-scale target detection model based on the network after upstream and downstream knowledge distillation.

8. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the lightweight industrial glass defect detection method based on weighted mask distillation as described in any one of claims 1 to 6 when executing the executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the lightweight industrial glass defect detection method based on weighted mask distillation as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the lightweight industrial glass defect detection method based on weighted mask distillation described in any one of claims 1 to 6 is implemented.

Images