Unsupervised image defect detection method, device and medium based on knowledge distillation
By constructing a teacher-student model system and using domain-agnostic datasets to train the teacher model and transfer knowledge, the problem of insufficient defect samples in manual visual inspection and supervised learning models is solved, and unsupervised high-precision image defect detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 四川启睿克科技有限公司
- Filing Date
- 2022-06-23
- Publication Date
- 2026-05-26
Smart Images
Figure CN115187525B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of image processing and machine learning, and in particular to an unsupervised image defect detection method, apparatus and medium based on knowledge distillation. Background Technology
[0002] The Industrial Internet is a new industrial ecosystem, key infrastructure, and novel application model that deeply integrates next-generation information and communication technologies with the industrial economy. Based on networks, centered on platforms, driven by data, and guaranteed by security, it comprehensively connects people, machines, and things, transforming traditional manufacturing models, production organization methods, and industrial forms. It constructs a new industrial production, manufacturing, and service system that is fully interconnected across all factors, the entire industrial chain, the entire value chain, and is of great significance for supporting the construction of a manufacturing powerhouse and a cyber powerhouse, enhancing the modernization level of the industrial chain, promoting high-quality economic development, and building a new development pattern.
[0003] In the Industrial Internet, preventative product inspection is a crucial component. It enables the timely detection of product defects and their evolving trends, and, combined with real-time process parameters, optimizes production processes to reduce defect rates, lower production costs, and increase company profits. Preventative product inspection primarily consists of three stages: real-time process parameter acquisition, product defect detection, and closed-loop process optimization.
[0004] In the product defect detection stage, the industry practice is to use visual inspection by professionals or to use a defect detection model based on supervised learning.
[0005] The following problems exist with the use of manual visual inspection: (1) Professional visual inspection personnel need to be trained; (2) Manual visual inspection is subjective to a certain extent, and the judgment ability of visual inspectors will be affected by their own emotions and physical condition; (3) Manual visual inspection is a post-processing and has no positive impact on the optimization of the entire process in real time.
[0006] Currently, the defect detection model based on supervised learning has the following problems: (1) The learning process is cumbersome; (2) It cannot learn when there are few or no defect samples. Summary of the Invention
[0007] This invention provides an unsupervised image defect detection method, apparatus, and medium based on knowledge distillation to solve the above-mentioned technical problems.
[0008] The technical solution adopted in this invention is: to provide an unsupervised image defect detection method based on knowledge distillation, comprising:
[0009] A teacher model is constructed, which is trained using a dataset that is not related to the domain of the defect to be detected, and then provides knowledge guidance that is beneficial to defect detection to the student model.
[0010] Construct a student model, which is used to extract multi-scale feature maps of images;
[0011] A multi-scale feature set of defect-free images is constructed using a student model, resulting in a set of student feature maps;
[0012] A set of pixel-level defect detection models is established using the student feature map set;
[0013] Receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal;
[0014] The student model is used to build a multi-scale feature map of the image. The distance between the multi-scale feature map and the pixel-level defect detection model set is calculated to obtain a distance matrix. By using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold, it is determined whether the image sent by the user terminal has defects.
[0015] Furthermore, the method for constructing the teacher model includes:
[0016] A neural network model is constructed as the initial teacher model, along with a dataset that is not related to the domain of the defect to be detected. The reason for constructing a dataset that is not related to the domain of the defect is that collecting defect samples in the domain generally requires a lot of manpower and resources, while this embodiment uses a large number of labeled samples from other domains as training samples, thereby greatly saving manpower and resources.
[0017] The loss function is set as cross-entropy;
[0018] The initial teacher model is trained using backpropagation on a dataset that is not domain-independent of the defect to be detected, thus obtaining the teacher model.
[0019] Furthermore, the method for constructing the student model includes:
[0020] Construct an initial student model and a defect-free dataset in the domain of defects to be detected;
[0021] Set the loss function as In the formula, L represents the number of layers in the intermediate network, where l∈[1,L], w l The weights of the l-th layer of the network are... This represents the local error of the l-th layer in both the teacher and student networks. Let α represent the global error of the l-th layer of the teacher network and the student network, where α∈[0,1] is the loss to balance the global and local errors. totalThis represents the total knowledge bias between the teacher network and the student network. Through this bias, the student and teacher networks are connected, and knowledge from the teacher network is transferred to the student network by minimizing the objective function. and The feature map of the teacher network is obtained as follows: The pooling characteristics of teacher networks are The characteristic graph of the student network is The pooling characteristics of student networks are Define local error Global error Where Disy(.) indicates that the distance metric includes at least Euclidean distance or cosine distance;
[0022] The initial student model is trained using backpropagation on a defect-free dataset in the domain of defects to be detected. During the training process, the parameters of the teacher model remain unchanged, and only the parameters of the initial student model are updated to obtain the student model.
[0023] Furthermore, the method for constructing the student feature map set includes:
[0024] Input a defect-free image into the student model and extract the output feature map of at least one intermediate layer;
[0025] Upsample the low-resolution feature map to the same size as the maximum resolution feature map in the output feature map;
[0026] Then all the feature maps are stitched together along the channel direction according to pixel positions;
[0027] The feature vector corresponding to each pixel in the stitched feature map is normalized using the L2 norm along the feature channel direction.
[0028] Furthermore, the method for establishing a set of pixel-level defect detection models includes:
[0029] The feature vectors corresponding to the same pixel points in all feature maps in the student feature map set are used as the feature set of the current pixel point;
[0030] A Gaussian mixture model is used to establish a feature distribution model for the feature set of each pixel.
[0031] The feature distribution model of all pixels is used as a set of pixel-level defect detection models.
[0032] Furthermore, the method for constructing the multi-scale feature map of the image using the student model includes:
[0033] Extract multi-scale feature maps of images sent by user terminals using a student model;
[0034] Upsample the small-resolution feature map in the multi-scale feature map to the same size as the maximum resolution;
[0035] All feature maps are stitched together along the feature channel direction according to pixel position;
[0036] The concatenated feature map is normalized by L2 norm along the feature channel direction to obtain the multi-scale feature map of the image.
[0037] Furthermore, the method for obtaining the distance matrix includes:
[0038] The similarity between the feature vector of each pixel in the multi-scale feature map of the image sent by the user terminal and the Gaussian distribution model corresponding to each pixel in the pixel-level defect detection model set is calculated to obtain the distance of each pixel. The distances of all pixels are combined into a distance matrix.
[0039] Upsample the distance matrix to the size of the original input image to obtain a distance matrix with the same size as the input image.
[0040] The distance matrix is obtained by smoothing it using filtering.
[0041] Furthermore, the method for determining whether an image sent by a user terminal has defects includes:
[0042] If the value at a certain position in the distance matrix is greater than the pixel-level defect threshold, then there is a defect at the corresponding position in the image; otherwise, there is no defect.
[0043] If the maximum value in the distance matrix is greater than the image-level defect threshold, then the image has a defect; otherwise, it has no defect.
[0044] The present invention also provides an unsupervised image defect detection device based on knowledge distillation, comprising:
[0045] The teacher model building module is used to acquire teacher models, which provide knowledge guidance to student models that are helpful for defect detection.
[0046] The student model building module is used to obtain student models, and the student models extract multi-scale feature maps from images;
[0047] The feature map extraction module is used to construct a multi-scale feature set of defect-free images through the student model, thereby obtaining a set of student feature maps.
[0048] The defect detection model building module is used to build a set of pixel-level defect detection models from a set of student feature maps.
[0049] The receiving module is used to receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal.
[0050] The defect detection module uses the student model to build a multi-scale feature map of the image, calculates the distance between the multi-scale feature map and the pixel-level defect detection model set to obtain a distance matrix, and determines whether the image sent by the user terminal has defects by using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold.
[0051] The present invention also provides a storage medium storing computer instructions, which, when invoked, are used to execute the above-described unsupervised image defect detection method based on knowledge distillation.
[0052] The beneficial effects of this invention are: This invention adopts teacher-student model learning technology, which only requires defect-free samples to solve the problems of insufficient trap samples and defect-free samples, thus it is more universal, the features learned by the network are more robust, and the recognition accuracy is higher. Attached Figure Description
[0053] Figure 1 This is a schematic flowchart of the unsupervised image defect detection method based on knowledge distillation disclosed in this invention. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0055] Example 1:
[0056] See Figure 1 This embodiment presents an unsupervised image defect detection method based on knowledge distillation, comprising the following steps:
[0057] S1. Construct a teacher model, which is trained using a dataset that is not related to the domain of the defect to be detected, and then provides knowledge guidance to the student model that is beneficial to defect detection.
[0058] S2. Construct a student model, which is used to extract multi-scale feature maps of images;
[0059] S3. Construct a multi-scale feature set of defect-free images using the student model to obtain the student feature map set;
[0060] S4. Use the student feature map set to establish a set of pixel-level defect detection models;
[0061] S5. Receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal;
[0062] S6. Using the student model, establish a multi-scale feature map of the image, calculate the distance between the multi-scale feature map and the pixel-level defect detection model set to obtain a distance matrix, and determine whether the image sent by the user terminal has defects by using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold.
[0063] Specifically, the teacher model should include at least a convolutional neural network model or a recurrent neural network model.
[0064] In S1, the method for constructing the teacher model includes:
[0065] S101. Construct a neural network model as the initial teacher model and a dataset that is independent of the domain of the defect to be detected;
[0066] S102. Set the loss function to cross-entropy;
[0067] S103. The initial teacher model is trained on a dataset that is not related to the domain of the defect to be detected using backpropagation to obtain the teacher model.
[0068] Specifically, in S2, the method for constructing the student model includes:
[0069] S201. Construct an initial student model and a defect-free dataset for the domain of defects to be detected;
[0070] S202, Set the loss function as follows In the formula, L represents the number of layers in the intermediate network, where l∈[1,L], w l The weights of the l-th layer of the network are... This represents the local error of the l-th layer in both the teacher and student networks. Let α represent the global error of the l-th layer of the teacher network and the student network, where α∈[0,1] is the loss to balance the global and local errors. total This represents the total knowledge bias between the teacher network and the student network. Through this bias, the student and teacher networks are connected, and knowledge from the teacher network is transferred to the student network by minimizing the objective function. and The feature map of the teacher network is obtained as follows: The pooling characteristics of teacher networks are The characteristic graph of the student network is The pooling characteristics of student networks are Define local error Global error Where Dist(.) indicates that the distance metric includes at least Euclidean distance or cosine distance;
[0071] S203. The initial student model is trained on a defect-free dataset in the domain of defects to be detected using backpropagation. During the training process, the parameters of the teacher model remain unchanged, and only the parameters of the initial student model are updated to obtain the student model.
[0072] Specifically, in S3, the method for constructing the student feature map set includes:
[0073] S301. Input the defect-free image into the student model and extract the output feature map of at least one intermediate layer.
[0074] S302. Upsample the low-resolution feature map to a size consistent with the maximum resolution of the output feature map;
[0075] S303. Then, all the feature maps are stitched together along the channel direction according to the pixel positions.
[0076] S304. Normalize the feature vector corresponding to each pixel in the spliced feature map along the feature channel direction using the L2 norm.
[0077] Specifically, in S4, the method for establishing a set of pixel-level defect detection models includes:
[0078] S401. Take the feature vectors corresponding to the same pixel points in all feature map spectra in the student feature map set as the feature set of the current pixel point;
[0079] S402. Establish a feature distribution model for the feature set of each pixel using a Gaussian mixture model;
[0080] S403. Use the feature distribution model of all pixels as a set of pixel-level defect detection models.
[0081] Specifically, in S6, the method for establishing the multi-scale feature map of the image using the student model includes:
[0082] S601. Use the student model to extract multi-scale feature maps of images sent by user terminals;
[0083] S602. Upsample the small-resolution feature map in the multi-scale feature map to a size consistent with the maximum resolution.
[0084] S603. Stitch together all feature maps along the feature channel direction according to pixel position;
[0085] S604. Normalize the feature vector corresponding to each pixel along the feature channel direction of the stitched feature map to obtain the multi-scale feature map of the image.
[0086] Specifically, in S6, the method for obtaining the distance matrix includes:
[0087] S611. Calculate the similarity between the feature vector of each pixel in the multi-scale feature map of the image sent by the user terminal and the Gaussian distribution model corresponding to each pixel in the pixel-level defect detection model set to obtain the distance of each pixel. Combine the distances of all pixels into a distance matrix.
[0088] S612. Upsample the distance matrix to the size of the original input image to obtain a distance matrix with the same size as the input image.
[0089] S613. The distance matrix is smoothed by filtering to obtain the distance matrix.
[0090] Specifically, in S6, the method for determining whether an image sent by the user terminal has defects includes:
[0091] S621. If the value at a certain position in the distance matrix is greater than the pixel-level defect threshold, then there is a defect at the corresponding position in the image; otherwise, there is no defect.
[0092] S622. If the maximum value in the distance matrix is greater than the image-level defect threshold, then the image has a defect; otherwise, it has no defect.
[0093] Example 2
[0094] This embodiment discloses an unsupervised image defect detection device based on knowledge distillation, comprising:
[0095] The teacher model building module is used to acquire teacher models, which provide knowledge guidance to student models that are helpful for defect detection.
[0096] The student model building module is used to obtain student models, and the student models extract multi-scale feature maps from images;
[0097] The feature map extraction module is used to construct a multi-scale feature set of defect-free images through the student model, thereby obtaining a set of student feature maps.
[0098] The defect detection model building module is used to build a set of pixel-level defect detection models from a set of student feature maps.
[0099] The receiving module is used to receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal.
[0100] The defect detection module uses the student model to build a multi-scale feature map of the image, calculates the distance between the multi-scale feature map and the pixel-level defect detection model set to obtain a distance matrix, and determines whether the image sent by the user terminal has defects by using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold.
[0101] Example 3
[0102] This embodiment discloses a storage medium storing computer instructions. When the computer instructions are invoked, they are used to execute the knowledge distillation-based unsupervised image defect detection method described in Embodiment 1.
[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An unsupervised image defect detection method based on knowledge distillation, characterized in that, include: A teacher model is constructed, which is trained using a dataset that is not related to the domain of the defect to be detected, and then provides knowledge guidance that is beneficial to defect detection to the student model. The methods for constructing the teacher model include: Construct a neural network model as the initial teacher model and a dataset that is not related to the domain of the defect to be detected; The loss function is set as cross-entropy; The initial teacher model is trained using backpropagation on the dataset that is not related to the domain of the defect to be detected, thus obtaining the teacher model; Construct a student model, which is used to extract multi-scale feature maps of images; The method for constructing the student model includes: Construct an initial student model and a defect-free dataset in the domain of defects to be detected; Set the loss function as In the formula, L represents the number of layers in the intermediate network, where w l The weights of the l-th layer of the network are... The first part represents the teacher network and the student network. Local errors of the layer The first part represents the teacher network and the student network. Global error of the layer To balance global and local errors, This represents the total knowledge bias between the teacher network and the student network. Through this bias, the student and teacher networks are connected, and knowledge from the teacher network is transferred to the student network by minimizing the objective function. and The feature map of the teacher network is obtained as follows: The pooling characteristics of the teacher network are The characteristic graph of the student network is The pooling characteristics of student networks are Define local error global error ,in, This indicates that the distance metric includes at least Euclidean distance or cosine distance; The initial student model is trained using backpropagation on a defect-free dataset in the domain of defects to be detected. During the training process, the parameters of the teacher model remain unchanged, and only the parameters of the initial student model are updated to obtain the student model. A multi-scale feature set of defect-free images is constructed using a student model, resulting in a set of student feature maps; A set of pixel-level defect detection models is established using the student feature map set; Receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal; The student model is used to build a multi-scale feature map of the image. The distance between the multi-scale feature map and the pixel-level defect detection model set is calculated to obtain a distance matrix. By using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold, it is determined whether the image sent by the user terminal has defects.
2. The unsupervised image defect detection method based on knowledge distillation according to claim 1, characterized in that, The method for constructing the student feature map set includes: Input a defect-free image into the student model and extract the output feature map of at least one intermediate layer; Upsample the low-resolution feature map to the same size as the maximum resolution feature map in the output feature map; Then all the feature maps are stitched together along the channel direction according to pixel positions; The feature vector corresponding to each pixel in the stitched feature map is normalized using the L2 norm along the feature channel direction.
3. The unsupervised image defect detection method based on knowledge distillation according to claim 2, characterized in that, The method for establishing a set of pixel-level defect detection models includes: The feature vectors corresponding to the same pixel points in all feature maps in the student feature map set are used as the feature set of the current pixel point; A Gaussian mixture model is used to establish a feature distribution model for the feature set of each pixel. The feature distribution model of all pixels is used as a set of pixel-level defect detection models.
4. The unsupervised image defect detection method based on knowledge distillation according to claim 3, characterized in that, The method for constructing a multi-scale feature map of the image using the student model includes: Extract multi-scale feature maps of images sent by user terminals using a student model; Upsample the small-resolution feature map in the multi-scale feature map to the same size as the maximum resolution; All feature maps are stitched together along the feature channel direction according to pixel position; The concatenated feature map is normalized by L2 norm along the feature channel direction to obtain the multi-scale feature map of the image.
5. The unsupervised image defect detection method based on knowledge distillation according to claim 4, characterized in that, The method for obtaining the distance matrix includes: The similarity between the feature vector of each pixel in the multi-scale feature map of the image sent by the user terminal and the Gaussian distribution model corresponding to each pixel in the pixel-level defect detection model set is calculated to obtain the distance of each pixel. The distances of all pixels are combined into a distance matrix. Upsample the distance matrix to the size of the original input image to obtain a distance matrix with the same size as the input image. The distance matrix is obtained by smoothing it using filtering.
6. The unsupervised image defect detection method based on knowledge distillation according to claim 5, characterized in that, The method for determining whether an image sent by a user terminal has defects includes: If the value at a certain position in the distance matrix is greater than the pixel-level defect threshold, then there is a defect at the corresponding position in the image; otherwise, there is no defect. If the maximum value in the distance matrix is greater than the image-level defect threshold, then the image has a defect; otherwise, it has no defect.
7. An unsupervised image defect detection device based on knowledge distillation, characterized in that, include: The teacher model building module is used to acquire teacher models, which provide knowledge guidance to student models that are helpful for defect detection. The student model building module is used to obtain student models. Obtaining student models specifically includes the following steps: Construct an initial student model and a defect-free dataset in the domain of defects to be detected; Set the loss function as In the formula, L represents the number of layers in the intermediate network, where w l The weights of the l-th layer of the network are... The first part represents the teacher network and the student network. Local errors of the layer The first part represents the teacher network and the student network. Global error of the layer To balance global and local errors, This represents the total knowledge bias between the teacher network and the student network. Through this bias, the student and teacher networks are connected, and knowledge from the teacher network is transferred to the student network by minimizing the objective function. and The feature map of the teacher network is obtained as follows: The pooling characteristics of the teacher network are The characteristic graph of the student network is The pooling characteristics of student networks are Define local error global error ,in, This indicates that the distance metric includes at least Euclidean distance or cosine distance; The initial student model is trained using backpropagation on a defect-free dataset in the domain of defects to be detected. During the training process, the parameters of the teacher model remain unchanged, and only the parameters of the initial student model are updated to obtain the student model. Student models extract multi-scale feature maps from images; The feature map extraction module is used to construct a multi-scale feature set of defect-free images through the student model, thereby obtaining a set of student feature maps. The defect detection model building module is used to build a set of pixel-level defect detection models from a set of student feature maps. The receiving module is used to receive images, pixel-level defect thresholds, and image-level defect thresholds sent by the user terminal. The defect detection module uses the student model to build a multi-scale feature map of the image, calculates the distance between the multi-scale feature map and the pixel-level defect detection model set to obtain a distance matrix, and determines whether the image sent by the user terminal has defects by using the relationship between the distance matrix and the pixel-level defect threshold and the image-level defect threshold.
8. A storage medium, characterized in that, The storage medium stores computer instructions, which, when invoked, are used to execute the knowledge distillation-based unsupervised image defect detection method as described in any one of claims 1-5.