Mutual supervision defect detection method and device

Abstract

The invention discloses a mutual supervision defect detection method and device, and relates to the technical field of product defect detection, and the method comprises the steps: determining a defect candidate region according to a target product image; inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; and in a mutual supervision detection mode, inputting the defect candidate region into a depth feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determining a defect determination result according to a mutual supervision strategy. According to the invention, accurate defect determination can be carried out on the target product image.

Description

Technical Field

[0001] This application relates to the technical field of product defect detection, and particularly to a mutual supervision defect detection method and device. Background Art

[0002] In the field of industrial inspection, automatic identification of product appearance defects through visual inspection mainly has two methods at present: 1. Traditional image algorithms; 2. Deep learning technology. The generalization ability of traditional image algorithms is poor, and they are sensitive to factors such as the brightness of images, noise, and the contrast of defects. Therefore, they have high requirements for product consistency and rely relatively heavily on the experience of engineers. In deep learning technology, the construction of deep networks still relies on experience, depends on a large amount of labeled data and learning samples, has high requirements for hardware, and it is difficult to effectively trace the over-inspection and missed-inspection situations of products; the results are relatively difficult to effectively quantify and cannot meet the variable detection requirements on-site. It can be seen that both of these two defect detection methods have deficiencies.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a mutual supervision defect detection method and device, aiming to solve the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection.

[0005] To achieve the above purpose, this application proposes a mutual supervision defect detection method, and the mutual supervision defect detection method includes: Determine defect candidate regions based on the target product image; Input the defect candidate regions into a traditional feature filter to obtain a traditional output result based on prior knowledge; In the mutual supervision detection mode, input the defect candidate regions into a deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determine a defect determination result according to the mutual supervision strategy.

[0006] In one embodiment, the step of inputting the defect candidate regions into a traditional feature filter to obtain a traditional output result based on prior knowledge includes: Extract each connected component of the defect candidate regions; Extract features from each connected component based on a target feature value set to obtain a feature value set; Based on the feature value set and the target feature value set, determine the defect regions and normal regions in the defect candidate regions; Determine a traditional output result according to the defect regions and the normal regions.

[0007] In one embodiment, in the mutual supervision detection mode, the step of inputting the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result, and determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: Determine the determination preference value of the target feature value set; If the determination preference value is the first value, after entering the non-mutual supervision detection mode, use the traditional output result as the defect determination result, and use the traditional initial result as the defect sample to train the deep feature extraction network; If the determination preference value is the second value, after entering the mutual supervision detection mode, input the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result, and determine the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0008] In one embodiment, the step of inputting the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result includes: Input the defect candidate region into the deep feature extraction network to obtain defect candidate region feature information; Determine the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database, and use the maximum cosine similarity as the output score; Determine the artificial intelligence output result based on the comparison result between the output score and the matching score threshold.

[0009] In one embodiment, the deep feature extraction network includes a shape weight correction module and a shape attention module. The shape weight correction module includes an SW coefficient layer, a max pooling layer, an average pooling layer, and a splicing layer. The shape attention module includes a channel attention layer and a defect shape weight attention layer. Among them, the step of inputting the defect candidate region into the deep feature extraction network to obtain defect candidate region feature information includes: Input the defect candidate region into the shape weight correction module. Determine the first extraction feature through the SW coefficient layer and the max pooling layer, determine the second extraction feature through the average pooling layer, and splice the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer; Input the third extraction feature into the shape attention module, and determine the defect candidate region feature information through the channel attention layer and the defect shape weight attention layer.

[0010] In one embodiment, the step of determining the first extracted feature through the SW coefficient layer and the max pooling layer includes: Obtain the original defect image; Determine the image mean of the original defect image according to the saliency algorithm; Based on the original defect image, the image mean, and the stretching coefficient, determine the SW coefficients of the SW coefficient layer; Process the defect candidate region through the SW coefficients of the SW coefficient layer to obtain the initial extracted feature, and process the initial extracted feature through the max pooling layer to obtain the first extracted feature.

[0011] In one embodiment, the step of determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: Determine whether the traditional output result is the same as the artificial intelligence output result; When it is determined that the traditional output result is the same as the artificial intelligence output result, use the same result as the defect determination result; When it is determined that the traditional output result is different from the artificial intelligence output result, determine the feature confidence of the target feature value set, and determine the defect determination result according to the feature confidence.

[0012] In one embodiment, the step of determining the defect determination result according to the feature confidence includes: When the feature confidence is the first value and the traditional output result is a defect region, use the traditional output result as the defect determination result, and use the traditional output result as a defect sample to retrain the deep feature extraction network; When the feature confidence is the first value and the traditional output result is a normal region, use the artificial intelligence output result as the defect determination result, and update the screening threshold of the target feature value set; When the feature confidence is the second value and the traditional output result is a defect region, use the artificial intelligence output result as the defect determination result; When the feature confidence is the second value and the traditional output result is a normal region, use the traditional output result as the defect determination result, and use the traditional output result as a defect sample to retrain the deep feature extraction network.

[0013] In one embodiment, the step of determining the defect candidate region according to the target product image includes: Perform image preprocessing on the target product image to obtain a preprocessed product image, where the image preprocessing includes image denoising and contrast enhancement; Perform defect filtering on the preprocessed product image to obtain a defect-filtered image; Perform binarization processing on the defect-filtered image to obtain defect candidate regions.

[0014] In addition, to achieve the above object, the present application also proposes a mutual-supervision defect detection device, and the mutual-supervision defect detection device includes: A determination module, configured to determine defect candidate regions according to a target product image; An input module, configured to input the defect candidate regions into a traditional feature filter to obtain a traditional output result based on prior knowledge; The determination module is further configured to, in a mutual-supervision detection mode, input the defect candidate regions into a deep feature extraction network to obtain an artificial intelligence output result, and determine a defect determination result according to a mutual-supervision strategy based on the traditional output result and the artificial intelligence output result.

[0015] In addition, to achieve the above object, the present application also proposes a mutual-supervision defect detection device, and the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the mutual-supervision defect detection method as described above.

[0016] In addition, to achieve the above object, the present application also proposes a storage medium, and the storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the mutual-supervision defect detection method as described above are implemented.

[0017] In addition, to achieve the above object, the present application also provides a computer program product, and the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the mutual-supervision defect detection method as described above are implemented.

[0018] One or more technical solutions proposed by the present application have at least the following technical effects: The mutual - supervised defect detection method and device proposed in this application determine defect candidate regions based on the target product image; input the defect candidate regions into a traditional feature filter to obtain a traditional output result based on prior knowledge; in the mutual - supervised detection mode, input the defect candidate regions into a deep feature extraction network to obtain an artificial intelligence output result, and determine a defect judgment result according to the mutual - supervised strategy based on the traditional output result and the artificial intelligence output result. This solves the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, this application can combine a traditional feature filter and a deep feature extraction network to jointly perform defect judgment on the target product image, and improve the defect recognition capabilities of the traditional feature filter and the deep feature extraction network respectively through mutual - supervised learning, so as to achieve accurate defect judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic flowchart provided for the first embodiment of the mutual - supervised defect detection method of this application; Figure 2 It is a schematic diagram of the mutual - supervised strategy provided for the first embodiment of the mutual - supervised defect detection method of this application; Figure 3 It is a schematic flowchart provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 4 It is a schematic diagram of the extraction process of the first extracted feature provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 5 It is a schematic diagram of the extraction process of the second extracted feature provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 6 It is a schematic diagram of the extraction process of the third extracted feature provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 7 It is a schematic diagram of the shape attention module provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 8 It is a schematic diagram of the defect - shape weight attention layer provided for the second embodiment of the mutual - supervised defect detection method of this application; Figure 9 Schematic diagram of the deep feature extraction network provided for the second embodiment of the mutual supervision defect detection method of this application; Figure 10 Schematic diagram of the construction of the SW coefficient provided for the second embodiment of the mutual supervision defect detection method of this application; Figure 11 Schematic diagram of the module structure of the mutual supervision defect detection device of the embodiment of this application; Figure 12 Schematic diagram of the device structure of the hardware operating environment involved in the mutual supervision defect detection method in the embodiment of this application.

[0022] The realization of the purpose, functional features and advantages of this application will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0023] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0024] In order to better understand the technical solutions of this application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0025] The main solution of the embodiment of this application is: determining a defect candidate area according to the target product image; inputting the defect candidate area into a traditional feature filter to obtain a traditional output result based on prior knowledge; in the mutual supervision detection mode, inputting the defect candidate area into a deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determining a defect determination result according to the mutual supervision strategy.

[0026] As can be seen from the above embodiments, this application determines a defect candidate area according to the target product image; inputs the defect candidate area into a traditional feature filter to obtain a traditional output result based on prior knowledge; in the mutual supervision detection mode, inputs the defect candidate area into a deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determines a defect determination result according to the mutual supervision strategy, solving the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, this application can combine a traditional feature filter and a deep feature extraction network to jointly perform defect determination on the target product image, and respectively improve the defect recognition capabilities of the traditional feature filter and the deep feature extraction network through mutual supervision learning, so as to achieve accurate defect determination.

[0027] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a mutual supervision defect detection device, etc. that can implement the above functions. Taking mutual supervision defect detection as an example, this embodiment and the following embodiments will be described below.

[0028] Based on this, an embodiment of the present application provides a mutual supervision defect detection method, referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the mutual supervision defect detection method of the present application.

[0029] In this embodiment, the mutual supervision defect detection method includes steps S10 to S30: Step S10, determining a defect candidate region according to the target product image; In a feasible implementation manner, the step of determining a defect candidate region according to the target product image includes: performing image preprocessing on the target product image to obtain a preprocessed product image, where the image preprocessing includes image denoising and contrast enhancement; performing defect filtering on the preprocessed product image to obtain a defect-filtered image; performing binarization processing on the defect-filtered image to obtain a defect candidate region.

[0030] It should be noted that the target product image can be obtained through an imaging system, and the imaging system includes industrial cameras (including but not limited to area array cameras, line scan cameras, etc.), industrial lenses (including but not limited to telephoto lenses, short focal length lenses, telecentric lenses, zoom lenses, etc.), and vision light sources (including but not limited to bar light sources, ring light sources, coaxial light sources, etc.).

[0031] It should be noted that the image preprocessing mainly includes two functions, namely image denoising and contrast enhancement. Among them, image denoising includes but is not limited to Gaussian filtering, median filtering, mean filtering, bilateral filtering, etc.; contrast enhancement includes but is not limited to histogram equalization, histogram stretching, power transformation, logarithmic transformation, etc.; defect filtering can be implemented through a dot filter, a line filter, or a cluster filter (specifically refer to the patent "Defect Detection Method, Device, Defect Detection Equipment, and Computer Storage Medium"); defect segmentation refers to performing binarization processing on the defect filtering result, and defect segmentation methods include but are not limited to threshold segmentation method, OTSU method, global threshold method, adaptive threshold method, etc.

[0032] Step S20, inputting the defect candidate region into a traditional feature selector to obtain a traditional output result based on prior knowledge; In a feasible implementation manner, the step of inputting the defect candidate region into the traditional feature filter to obtain the traditional output result based on prior knowledge includes: extracting each connected component of the defect candidate region; performing feature extraction on each connected component based on the target feature value set to obtain a feature value set; determining the defect region and the normal region in the defect candidate region based on the feature value set and the target feature value set; and determining the traditional output result according to the defect region and the normal region.

[0033] It should be noted that for the segmented binary image B(x, y), each connected component {C1, C2, C3, …, C n} is extracted, and feature extraction is performed on each connected component (including but not limited to various features such as the size, length, width, contrast, rectangularity, circularity, aspect ratio, etc. of the region). Let the feature set of the connected component C m be {F1, F2, F3, …, F k}. According to the known defect features (i.e., the target feature value set), feature screening is performed on each connected component, and finally the defect region is obtained. The feature screening is carried out according to the following principle:

[0034] where i = (1, 2, 3, … k), T1 is the minimum threshold set for each feature in the target feature value set, T2 is the maximum threshold set for each feature in the target feature value set, and the determination condition for C m to be a defect region is:

[0035] It should be noted that when D m is equal to 1, it indicates that the region C m is a defect region (i.e., each feature screened out is within the feature range set for each feature in the target feature value set), otherwise it is a normal region.

[0036] Step S30, in the mutual supervision detection mode, input the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result, and determine the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0037] It should be noted that the mutual supervision strategy refers to jointly performing defect determination by combining the traditional output result and the artificial intelligence output result. After entering the mutual supervision detection mode, it indicates that the deep feature extraction network at this time is a successfully trained network model and can be used for accurate feature extraction for the defect candidate region.

[0038] In a feasible implementation manner, in the mutual supervision detection mode, the steps of inputting the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result, and determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result include: determining the determination preference value of the target feature value set; if the determination preference value is the first value, after entering the non-mutual supervision detection mode, taking the traditional output result as the defect determination result and taking the traditional initial result as the defect sample to train the deep feature extraction network; if the determination preference value is the second value, after entering the mutual supervision detection mode, inputting the defect candidate region into the deep feature extraction network to obtain the artificial intelligence output result, and determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0039] It should be noted that since the deep learning network requires a large number of training samples, when the deep learning network runs in the initial stage or encounters new types of defects, it is unable to provide a large number of training samples to train the model, which will result in the inapplicability of the deep learning network in this case. Therefore, in response to these problems, a detection strategy based on the determination preference value is set as follows: In the process of feature screening, for each set of target feature value sets used for feature screening, a determination preference setting is added (the determination preference value can be used to determine which detection mode to enter. For example, when the determination preference value is 1, enter the non-mutual supervision detection mode, and when the determination preference value is 0, enter the mutual supervision detection mode) to distinguish whether the set of target feature value sets is independent of defect determination, so as to solve the problem of insufficient system samples. The determination preference value is defined as: State = {1, 0} 1. When State = 1 (that is, the preference determination value is the first value), only the defect regions screened out by the set of target feature value sets are output, the defect candidate regions are not sent to the AI network for discrimination, and the screened-out defect samples are saved. These defect samples can be used to train the deep learning model (that is, the deep feature extraction network). Therefore, it is possible to maintain a certain level of defect detection (using traditional feature screening for defect detection) even without sufficient defect samples. 2. When State = 0 (that is, the preference determination value is the second value), enter the mutual supervision detection mode, and the defect determination result needs to be jointly determined by combining the traditional image algorithm (that is, the traditional feature filter) and the AI deep network (the deep feature extraction network).

[0040] In a feasible implementation manner, the step of determining a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: determining whether the traditional output result is the same as the artificial intelligence output result; when it is determined that the traditional output result is the same as the artificial intelligence output result, using the same result as the defect determination result; when it is determined that the traditional output result is different from the artificial intelligence output result, determining the feature confidence of the target feature value set, and determining the defect determination result according to the feature confidence.

[0041] It should be noted that when the traditional output result is the same as the artificial intelligence output result, the same result can be directly used as the defect determination result; when the traditional output result is different from the artificial intelligence output result, the defect determination result needs to be further determined according to the feature confidence.

[0042] In a feasible implementation manner, the step of determining a defect determination result according to the feature confidence includes: when the feature confidence is the first value and the traditional output result is a defect area, using the traditional output result as the defect determination result, and retraining the deep feature extraction network after using the traditional output result as a defect sample; when the feature confidence is the first value and the traditional output result is a normal area, using the artificial intelligence output result as the defect determination result, and updating the screening threshold of the target feature value set; when the feature confidence is the second value and the traditional output result is a defect area, using the artificial intelligence output result as the defect determination result; when the feature confidence is the second value and the traditional output result is a normal area, using the traditional output result as the defect determination result, and retraining the deep feature extraction network after using the traditional output result as a defect sample.

[0043] It should be noted that the first value is set to 1 and the second value is set to 0. As Figure 2 shown, when the traditional output result is the same as the artificial intelligence output result, the same result is directly used as the defect determination result; when the traditional output result is different from the artificial intelligence output result, the feature confidence of the target feature value set needs to be further determined.

[0044] In a specific implementation, when the feature confidence level is 1 and the traditional output result determines that the defect candidate region is a defect region, it indicates at this time that the artificial intelligence output result determines that the defect candidate region is a normal region (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence level is 1, it is very certain that the defect candidate region is a defect region. Therefore, the traditional output result can be used as the defect determination result. Since the artificial intelligence output result is inaccurate (indicating that the AI fails), the traditional output result can be saved as a defect sample and then used to update the AI model (i.e., the deep feature extraction network) later; when the feature confidence level is 1 and the traditional output result determines that the defect candidate region is a normal region, it indicates at this time that the artificial intelligence output result determines that the defect candidate region is a defect region (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence level is 1, the conditions for defect feature screening are relatively strict. Therefore, the artificial intelligence output result can be used as the defect determination result. Since the traditional output result determines that the defect candidate region is a normal region, it can be determined that the traditional feature filter used to output the traditional output result fails. At this time, the screening threshold of the target feature value set needs to be updated so that the traditional feature filter can ensure that the traditional output result is a defect region when the feature confidence level is 1.

[0045] It should be noted that in the process of feature screening, for each set of target feature value sets used for feature screening, a feature confidence level is added to distinguish the credibility of this set of feature sets for defect determination, so as to cooperate with the AI discrimination system for mutual supervision detection.

[0046] In a specific implementation, when the feature confidence level is 0 and the traditional output result determines that the defect candidate region is a defect region, it indicates at this time that the artificial intelligence output result determines that the defect candidate region is a normal region (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence level is 0, it is uncertain whether the defect candidate region is a defect region. Therefore, the artificial intelligence output result can be directly used as the defect determination result; when the feature confidence level is 0 and the traditional output result determines that the defect candidate region is a normal region, it indicates at this time that the artificial intelligence output result determines that the defect candidate region is a defect region (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence level is 0, the conditions for defect feature screening are relatively loose. Therefore, the system is more inclined to the candidate region being a normal region. Therefore, the traditional output result can be used as the defect determination result. Since the artificial intelligence output result is inaccurate (indicating that the AI fails), the traditional output result can be saved as a defect sample and then used to update the AI model (i.e., the deep feature extraction network) later. Define the confidence level as: Trust = {1, 0} 1. When Trust = 1 (i.e., the feature confidence is the first value), it indicates that for the area screened by this set of target feature value sets, the traditional algorithm is very certain that this area is a defective area. At this time, it tends to trust the results of the traditional feature filter. 2. When Trust = 0 (i.e., the feature confidence is the second value), it indicates that for the area screened by this set of target feature value sets, the algorithm is not sure that this area is a defective area. At this time, it tends to trust the output results of the deep feature extraction network.

[0047] In this embodiment, defective candidate areas are determined based on the target product image; the defective candidate areas are input into a traditional feature filter to obtain traditional output results based on prior knowledge; in the mutual supervision detection mode, the defective candidate areas are input into a deep feature extraction network to obtain artificial intelligence output results, and based on the traditional output results and the artificial intelligence output results, the defective determination results are determined according to the mutual supervision strategy, which solves the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, the present application can combine a traditional feature filter and a deep feature extraction network to jointly perform defective determination on the target product image, and improve the defect recognition capabilities of the traditional feature filter and the deep feature extraction network respectively through mutual supervision learning, so as to achieve accurate defective determination.

[0048] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 and step S30 further includes steps S301 to S303: Step S301, input the defective candidate area into the deep feature extraction network to obtain defective candidate area feature information; It should be noted that the deep feature extraction network can be used to accurately extract features for the defective candidate area.

[0049] Step S302, determine the maximum cosine similarity between the defective candidate area feature information and the defective features stored in the feature database, and use the maximum cosine similarity as the output score; It should be noted that some feature vectors (i.e., defective features) of defective areas are pre-stored in the feature database. After the deep feature extraction network extracts the corresponding defective candidate area feature information from the defective candidate area, the defective candidate area feature information is calculated for cosine similarity with the feature vectors stored in the feature database one by one, and the maximum cosine similarity is used as the output score.

[0050] Step S303, determine the artificial intelligence output result based on the comparison result between the output score and the matching score threshold.

[0051] It should be noted that the AI output result can be determined according to the comparison result between the output score and the matching score threshold. For example, when the output score is greater than or equal to the matching score threshold, it indicates that the manual output result determines that there is a defective area in the defective candidate area; when the output score is less than the matching score threshold, it indicates that the manual output result determines that the defective candidate area is a normal area.

[0052] In a feasible implementation manner, the depth feature extraction network includes a shape weight correction module and a shape attention module. The shape weight correction module includes an SW coefficient layer, a max pooling layer, an average pooling layer, and a splicing layer. The shape attention module includes a channel attention layer and a defective shape weight attention layer. Among them, the step of inputting the defective candidate area into the depth feature extraction network to obtain defective candidate area feature information includes: inputting the defective candidate area into the shape weight correction module, determining a first extraction feature through the SW coefficient layer and the max pooling layer, determining a second extraction feature through the average pooling layer, and splicing the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer; inputting the third extraction feature into the shape attention module, and determining defective candidate area feature information through the channel attention layer and the defective shape weight attention layer.

[0053] It should be noted that the depth feature extraction network is a more efficient deep neural network structure designed for defective shapes, which is used to accurately extract the feature expression of defects. In this network structure, we have designed two new plug-and-play modules: 1. Shape Weight Correction Module (SWCM). SWCM replaces the single pooling operation commonly used in the network structure and can solve the problem that the convolutional network lacks emphasis on learning the feature weight coefficients of target defects and background regions in global feature extraction. 2. Shape Block Attention Module (SBAM). SBAM introduces a defective shape attention module (i.e., a defective shape attention layer) on the basis of the Convolutional Block Attention Module (CBAM) to improve the model's ability to extract defective shape information. It should be noted that as Figure 4 shown, the new feature X1 (i.e., the first extraction feature) of the SW coefficient correction pooling and the new feature X2 (i.e., the second extraction feature) obtained by average pooling as Figure 5 shown are spliced to obtain as Figure 6The new output feature X3 (i.e., the third extracted feature) shown preserves both the global information of the input feature and the corrected detailed information, thus enriching the expressive power of the output features of the convolutional module.

[0054] It should be noted that, as Figure 7 shown, based on the design idea of CBAM, the shape attention module proposes the SBAM structure. On the basis of the CBAM structure, the channel attention layer (CAM, Channel Attention Module) is retained, and the spatial attention layer (SAM, Spatial Attention Module) is replaced by the defect shape weight attention layer (SWAM, ShapeWeight Attention Module).

[0055] It should be noted that, as Figure 8 shown, for the SWAM layer: a) For the input feature map, use the SW coefficient to correct the weight of each channel; b) Perform max-pooling and average-pooling operations on the corrected new feature map; c) Concatenate the max-pooling and average-pooling features, and then connect a convolutional layer for feature extraction to obtain a new feature with the channel compressed to 1, retaining the shape and spatial information; d) Perform a Sigmoid activation on the new feature to obtain the final shape weight attention feature; e) The SW coefficients in the SWAM module and the SW coefficients used in the SWCM module can be reused, so there is no additional consumption. This is because for an image, only one set of SW coefficients needs to be generated and can be used by different modules in the network, so there is no need to generate multiple sets of SW coefficients for different modules; In a specific implementation, as Figure 9 shown, the defect candidate region passes through the SWCM structure and the SBAM structure in sequence to obtain the network output (i.e., the defect candidate region feature information).

[0056] In a feasible implementation manner, the step of determining the first extracted feature through the SW coefficient layer and the max-pooling layer includes: obtaining the original defect image; determining the image mean of the original defect image according to the saliency algorithm; determining the SW coefficient of the SW coefficient layer based on the original defect image, the image mean, and the stretching coefficient; processing the defect candidate region through the SW coefficient of the SW coefficient layer to obtain the initial extracted feature, and processing the initial extracted feature through the max-pooling layer to obtain the first extracted feature.

[0057] It should be noted that there are two ways to construct the SW coefficients in the SW coefficient layer. One is to automatically calculate the significant features by an algorithm to generate coefficients, and the other is to generate them by manually annotating data.

[0058] In the specific implementation, as Figure 10 shown, the coefficients are generated by automatically calculating the significant features by an algorithm: using the significance algorithm to automatically calculate the proportion weight of the significant object in the whole image according to the input image:

[0059] where, I u represents the image mean value calculated by the significance algorithm, Iwh(x,y) represents the original image (i.e., the original defect image), α represents the stretching coefficient of the image mean value, and I w represents the coefficient map obtained by taking the absolute value of the distance between each pixel value of the original image calculated by the significance algorithm and the image mean value, and is used as the SW coefficient.

[0060] In the specific implementation, they are generated by manually annotating data: while manually annotating the defect image categories, manually annotating the weight coefficients of the defect regions. For example: annotating the defect regions as 1 and the background as 0 to generate a new binary image as the artificial SW coefficient.

[0061] It should be noted that after the SW coefficients are multiplied point by point with the input features (the convolutional layer features of the defect candidate regions) to obtain new features (i.e., the initial extracted features), a new pooled feature X1 (i.e., the first extracted feature) is obtained after one max pooling; since the initial extracted features are strengthened by the shape weight coefficients, the feature weights of the defect foreground regions are strengthened, while the feature weights of the background regions are suppressed.

[0062] It should be noted that the SW coefficients are coefficients determined during the network structure design and data annotation, so there will be no additional consumption in terms of coefficient generation during the inference process.

[0063] In this embodiment, by inputting the defect candidate regions into the deep feature extraction network, defect candidate region feature information is obtained; the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database is determined, and the maximum cosine similarity is used as the output score; based on the comparison result between the output score and the matching score threshold, the artificial intelligence output result is determined. In the above way, the feature expression of the defect can be accurately extracted through the deep feature extraction network to obtain a more accurate artificial intelligence output result.

[0064] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the mutual supervision defect detection method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0065] The present application also provides a mutual supervision defect detection device. Please refer to Figure 11 , the mutual supervision defect detection device includes: A determination module 10, configured to determine a defect candidate region according to a target product image; An input module 20, configured to input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; The determination module 10 is further configured to, in the mutual supervision detection mode, input the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result, and determine a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0066] The mutual supervision defect detection device provided by the present application adopts the mutual supervision defect detection method in the above embodiment, and can solve the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, the beneficial effects of the mutual supervision defect detection device provided by the present application are the same as those of the mutual supervision defect detection method provided by the above embodiment, and other technical features in the mutual supervision defect detection device are the same as those disclosed in the method of the above embodiment, and will not be elaborated herein.

[0067] The present application provides a mutual supervision defect detection device. The mutual supervision defect detection device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the mutual supervision defect detection method in the first embodiment above.

[0068] Next, refer to Figure 12 , which shows a schematic structural diagram of a mutual supervision defect detection device suitable for implementing the embodiments of the present application. The mutual supervision defect detection device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 12 The mutual supervision defect detection device shown is only an example and should not impose any limitation on the functions and usage scopes of the embodiments of the present application.

[0069] AsFigure 12 As shown, the mutual-supervision defect detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the mutual-supervision defect detection device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the mutual-supervision defect detection device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a mutual-supervision defect detection device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0070] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart may be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the method of the embodiments disclosed in the present application are executed.

[0071] The mutual-supervision defect detection device provided by the present application adopts the mutual-supervision defect detection method in the above embodiments, and can solve the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, the beneficial effects of the mutual-supervision defect detection device provided by the present application are the same as those of the mutual-supervision defect detection method provided by the above embodiments, and other technical features in the mutual-supervision defect detection device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0072] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0073] As mentioned above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0074] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the mutual supervision defect detection method in the above embodiments.

[0075] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0076] The above computer-readable storage medium can be included in the mutual supervision defect detection device; it can also exist separately without being assembled into the mutual supervision defect detection device.

[0077] The above computer-readable storage medium stores one or more programs, which, when executed by the mutual-supervision defect detection device, cause the mutual-supervision defect detection device to: determine a defect candidate region based on a target product image; input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; in the mutual-supervision detection mode, input the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determine a defect determination result according to a mutual-supervision strategy.

[0078] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0080] The modules described in the embodiments of the present application may be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0081] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned mutual supervision defect detection method, which can solve the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the mutual supervision defect detection method provided by the above embodiment, and will not be elaborated here.

[0082] This application also provides a computer program product, including a computer program, and the steps of the above-mentioned mutual supervision defect detection method are implemented when the computer program is executed by a processor.

[0083] The computer program product provided by this application can solve the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the mutual supervision defect detection method provided by the above embodiment, and will not be elaborated here.

[0084] The above are only partial embodiments of this application, and thus do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. A mutual - supervision defect detection method, characterized in that, The method includes: Determining a defect candidate region according to a target product image; Inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; In a mutual supervision detection mode, inputting the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result, and determining a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

2. The method according to claim 1, characterized in that, The step of inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge includes: Extracting each connected component of the defect candidate region; Performing feature extraction on each connected component based on a target feature value set to obtain a feature value set; Based on the feature value set and the target feature value set, determining a defect region and a normal region in the defect candidate region; Determining a traditional output result according to the defect region and the normal region.

3. The method according to claim 2, characterized in that, The step of, in a mutual supervision detection mode, inputting the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result, and determining a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: Determining a determination preference value of the target feature value set; If the determination preference value is a first value, after entering a non-mutual supervision detection mode, using the traditional output result as the defect determination result and using the traditional initial result as a defect sample to train the deep feature extraction network; If the determination preference value is a second value, after entering a mutual supervision detection mode, inputting the defect candidate region into the deep feature extraction network to obtain an artificial intelligence output result, and determining a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

4. The method according to claim 1 or 3, characterized in that, The step of inputting the defect candidate region into the deep feature extraction network to obtain an artificial intelligence output result includes: Inputting the defect candidate region into the deep feature extraction network to obtain defect candidate region feature information; Determining the maximum cosine similarity between the defect candidate region feature information and the defect features stored in a feature database, and using the maximum cosine similarity as an output score; Determining an artificial intelligence output result based on a comparison result between the output score and a matching score threshold.

5. The method according to claim 4, characterized in that, The deep feature extraction network includes a shape weight correction module and a shape attention module. The shape weight correction module includes an SW coefficient layer, a max pooling layer, an average pooling layer, and a splicing layer. The shape attention module includes a channel attention layer and a defect shape weight attention layer; wherein, the step of inputting the defect candidate region into the deep feature extraction network to obtain defect candidate region feature information includes: Inputting the defect candidate region into the shape weight correction module, determining a first extraction feature through the SW coefficient layer and the max pooling layer, determining a second extraction feature through the average pooling layer, and splicing the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer; Input the third extracted feature into the shape attention module, and determine the defect candidate region feature information through the channel attention layer and the defect shape weight attention layer.

6. The method according to claim 5, characterized in that, The step of determining the first extracted feature through the SW coefficient layer and the max pooling layer includes: Obtain the original defect image; Determine the image mean of the original defect image according to the saliency algorithm; Based on the original defect image, the image mean, and the stretching coefficient, determine the SW coefficient of the SW coefficient layer; Process the defect candidate region with the SW coefficient of the SW coefficient layer to obtain the initial extracted feature, and process the initial extracted feature through the max pooling layer to obtain the first extracted feature.

7. The method according to claim 2, characterized in that, The step of determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: Judge whether the traditional output result is the same as the artificial intelligence output result; When it is determined that the traditional output result is the same as the artificial intelligence output result, use the same result as the defect determination result; When it is determined that the traditional output result is different from the artificial intelligence output result, determine the feature confidence of the target feature value set, and determine the defect determination result according to the feature confidence.

8. The method according to claim 7, characterized in that, The step of determining the defect determination result according to the feature confidence includes: When the feature confidence is the first value and the traditional output result is a defect region, use the traditional output result as the defect determination result, and use the traditional output result as a defect sample to retrain the deep feature extraction network; When the feature confidence is the first value and the traditional output result is a normal region, use the artificial intelligence output result as the defect determination result, and update the screening threshold of the target feature value set; When the feature confidence is the second value and the traditional output result is a defect region, use the artificial intelligence output result as the defect determination result; When the feature confidence is the second value and the traditional output result is a normal region, use the traditional output result as the defect determination result, and use the traditional output result as a defect sample to retrain the deep feature extraction network.

9. The method according to claim 1, wherein The step of determining the defect candidate region according to the target product image includes: Perform image preprocessing on the target product image to obtain a preprocessed product image, where the image preprocessing includes image denoising and contrast enhancement; Perform defect filtering on the preprocessed product image to obtain a defect filtered image; Perform binarization processing on the defect filtered image to obtain a defect candidate region.

10. A mutual supervision defect detection device, wherein The device includes: A determination module, configured to determine a defect candidate region according to a target product image; An input module, configured to input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; The determination module is further configured to, in the mutual supervision detection mode, input the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result, and determine a defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

Images