Mutual supervision defect detection method and device

Abstract

This application discloses a mutual supervision defect detection method and device, which relates to the field of product defect detection technology, including: determining defect candidate areas based on target product images; inputting the defect candidate areas into a traditional feature filter to obtain traditional output results based on prior knowledge; in a mutual supervision detection mode, inputting the defect candidate areas into a deep feature extraction network to obtain artificial intelligence output results, and determining defect determination results based on the traditional output results and the artificial intelligence output results according to a mutual supervision strategy. This application can accurately determine defects in target product images.

Description

Technical Field

[0001] The present application relates to the technical field of product defect detection, and in particular to a mutual supervision defect detection method and device. Background Art

[0002] In the field of industrial inspection, there are currently two main methods for automatically identifying product defects through visual inspection: 1. Traditional image algorithms; 2. Deep learning technology. Traditional image algorithms have poor generalization capabilities and are sensitive to image brightness, noise, and defect contrast. Therefore, they require high product consistency and rely heavily on the engineer's experience. Deep learning technology still relies on experience to build deep networks, relying on large amounts of labeled data and learning samples. This places high demands on hardware, making it difficult to effectively trace product passes and omissions. Furthermore, the results are difficult to quantify, making them unable to meet the ever-changing inspection requirements of the field. Clearly, both defect detection methods have shortcomings.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a mutual supervision defect detection method and device, aiming to solve the technical problems of insufficient product defect detection using a single traditional image algorithm or deep learning technology.

[0005] To achieve the above objectives, the present application proposes a mutual supervision defect detection method, which includes:

[0006] Determine defect candidate areas based on the target product image;

[0007] Inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge;

[0008] In the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain the artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, the defect judgment result is determined according to the mutual supervision strategy.

[0009] In one embodiment, the step of inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge includes:

[0010] Extracting connected domains of the defect candidate area;

[0011] Based on the target eigenvalue set, features are extracted from each connected domain to obtain a eigenvalue set;

[0012] Determining a defect area and a normal area in the defect candidate area based on the feature value set and the target feature value set;

[0013] A conventional output result is determined according to the defective area and the normal area.

[0014] In one embodiment, the step of inputting the defect candidate region into a deep feature extraction network in the mutual supervision detection mode to obtain an artificial intelligence output result, and determining a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes:

[0015] Determining a judgment preference value of the target feature value set;

[0016] If the determination preference value is the first value, then after entering the non-mutually supervised detection mode, the traditional output result is used as the defect determination result, and the traditional initial result is used as the defect sample to train the deep feature extraction network;

[0017] If the judgment preference value is the second value, after entering the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain the artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, the defect judgment result is determined according to the mutual supervision strategy.

[0018] In one embodiment, the step of inputting the defect candidate region into a deep feature extraction network to obtain an artificial intelligence output result includes:

[0019] Inputting the defect candidate region into the deep feature extraction network to obtain feature information of the defect candidate region;

[0020] Determine the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database, and use the maximum cosine similarity as the output score;

[0021] An artificial intelligence output result is determined based on a comparison result of the output score and a matching score threshold.

[0022] In one embodiment, the deep feature extraction network includes a shape weight correction module and a shape attention module, the shape weight correction module includes an SW coefficient layer, a maximum pooling layer, a mean pooling layer, and a splicing layer, and the shape attention module includes a channel attention layer and a defect shape weight attention layer; wherein, the step of inputting the defect candidate area into the deep feature extraction network to obtain feature information of the defect candidate area includes:

[0023] Input the defect candidate area into the shape weight correction module, determine the first extraction feature through the SW coefficient layer and the maximum pooling layer, determine the second extraction feature through the mean pooling layer, and splice the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer;

[0024] The third extracted feature is input into the shape attention module, and the defect candidate area feature information is determined through the channel attention layer and the defect shape weight attention layer.

[0025] In one embodiment, the step of determining the first extracted feature through the SW coefficient layer and the maximum pooling layer includes:

[0026] Obtaining the original defect image;

[0027] Determining an image mean of the original defect image according to a saliency algorithm;

[0028] determining a SW coefficient of the SW coefficient layer based on the original defect image, the image mean, and a stretching coefficient;

[0029] The defect candidate area is processed by the SW coefficient of the SW coefficient layer to obtain an initial extraction feature, and the initial extraction feature is processed by the maximum pooling layer to obtain a first extraction feature.

[0030] In one embodiment, the step of determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes:

[0031] Determining whether the traditional output result is the same as the artificial intelligence output result;

[0032] When it is determined that the traditional output result and the artificial intelligence output result are the same, the same result is used as the defect determination result;

[0033] When it is determined that the traditional output result is different from the artificial intelligence output result, the feature confidence of the target feature value set is determined, and the defect determination result is determined based on the feature confidence.

[0034] In one embodiment, the step of determining the defect determination result according to the feature confidence level includes:

[0035] When the feature confidence is a first value and the traditional output result is a defect area, using the traditional output result as a defect determination result, and retraining the deep feature extraction network after using the traditional output result as a defect sample;

[0036] When the feature confidence is a first value and the traditional output result is in a normal area, taking the artificial intelligence output result as a defect determination result, and updating the screening threshold of the target feature value set;

[0037] When the feature confidence is a second value and the traditional output result is a defect area, using the artificial intelligence output result as a defect determination result;

[0038] When the feature confidence is the second value and the traditional output result is a normal area, the traditional output result is used as a defect determination result, and the deep feature extraction network is retrained after the traditional output result is used as a defect sample.

[0039] In one embodiment, the step of determining a candidate defect area based on the target product image includes:

[0040] Performing image preprocessing on the target product image to obtain a preprocessed product image, wherein the image preprocessing includes image denoising and contrast enhancement;

[0041] performing defect filtering on the preprocessed product image to obtain a defect filtered image;

[0042] Binarization is performed on the defect filtered image to obtain a defect candidate area.

[0043] In addition, to achieve the above objectives, the present application also proposes a mutual supervision defect detection device, which includes:

[0044] A determination module, configured to determine a defect candidate area based on a target product image;

[0045] An input module, configured to input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge;

[0046] The determination module is also used to input the defect candidate area into the deep feature extraction network in the mutual supervision detection mode to obtain the artificial intelligence output result, and determine the defect judgment result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a mutual supervision defect detection device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the mutual supervision defect detection method as described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the mutual supervision defect detection method described above are implemented.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the mutual supervision defect detection method described above.

[0050] One or more technical solutions proposed in this application have at least the following technical effects:

[0051] The mutual supervision defect detection method and device proposed in the present application determine the defect candidate area according to the target product image; input the defect candidate area into the traditional feature filter to obtain the traditional output result based on prior knowledge; in the mutual supervision detection mode, input the defect candidate area into the deep feature extraction network to obtain the artificial intelligence output result, and determine the defect judgment result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result, which solves the technical problems of insufficient product defect detection using a single traditional image algorithm or deep learning technology. Compared with the existing technology, the present application can combine the traditional feature filter and the deep feature extraction network to jointly perform defect judgment on the target product image, and improve the defect recognition capabilities of the traditional feature filter and the deep feature extraction network through mutual supervision learning, thereby achieving accurate defect judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 A flowchart of the first embodiment of the mutual supervision defect detection method of this application is provided;

[0055] Figure 2 A schematic diagram of a mutual supervision strategy provided in Example 1 of the mutual supervision defect detection method of this application;

[0056] Figure 3 A flowchart of the second embodiment of the mutual supervision defect detection method of this application is provided;

[0057] Figure 4 A schematic diagram of the extraction process of the first extracted feature provided in Example 2 of the mutual supervision defect detection method of this application;

[0058] Figure 5 A schematic diagram of the extraction process of the second extracted feature provided in Example 2 of the mutual supervision defect detection method of this application;

[0059] Figure 6 A schematic diagram of the extraction process of the third extracted feature provided in Example 2 of the mutual supervision defect detection method of this application;

[0060] Figure 7 Schematic diagram of the shape attention module provided for the second embodiment of the mutual supervision defect detection method of this application;

[0061] Figure 8 Schematic diagram of defect shape weight attention layer provided for embodiment 2 of the mutual supervision defect detection method of this application;

[0062] Figure 9 A schematic diagram of a deep feature extraction network provided for the second embodiment of the mutual supervision defect detection method of this application;

[0063] Figure 10 A schematic diagram of the construction of the SW coefficient provided in Example 2 of the mutual supervision defect detection method of this application;

[0064] Figure 11 This is a schematic diagram of the module structure of the mutual supervision defect detection device according to an embodiment of the present application;

[0065] Figure 12 Schematic diagram of the device structure of the hardware operating environment involved in the mutual supervision defect detection method in the embodiment of the present application.

[0066] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0068] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0069] The main solution of the embodiment of the present application is: determining the defect candidate area based on the target product image; inputting the defect candidate area into the traditional feature filter to obtain the traditional output result based on prior knowledge; in the mutual supervision detection mode, inputting the defect candidate area into the deep feature extraction network to obtain the artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determining the defect judgment result according to the mutual supervision strategy.

[0070] It can be seen from the above embodiments that the present application determines the defect candidate area according to the target product image; inputs the defect candidate area into the traditional feature filter to obtain the traditional output result based on prior knowledge; in the mutual supervision detection mode, inputs the defect candidate area into the deep feature extraction network to obtain the artificial intelligence output result, and determines the defect judgment result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result, thereby solving the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the existing technology, the present application can combine the traditional feature filter and the deep feature extraction network to jointly perform defect judgment on the target product image, and improve the defect recognition capabilities of the traditional feature filter and the deep feature extraction network through mutual supervision learning, thereby achieving accurate defect judgment.

[0071] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, a mutual supervision defect detection device, etc. The following uses mutual supervision defect detection as an example to illustrate this embodiment and the following embodiments.

[0072] Based on this, the embodiment of the present application provides a mutual supervision defect detection method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the mutual supervision defect detection method of this application.

[0073] In this embodiment, the mutual supervision defect detection method includes steps S10 to S30:

[0074] Step S10, determining defect candidate areas based on the target product image;

[0075] In a feasible embodiment, the step of determining the defect candidate area based on the target product image includes: performing image preprocessing on the target product image to obtain a preprocessed product image, wherein the image preprocessing includes image denoising and contrast enhancement; performing defect filtering on the preprocessed product image to obtain a defect filtered image; and performing binarization processing on the defect filtered image to obtain the defect candidate area.

[0076] It should be noted that the target product image can be obtained through an imaging system, which includes industrial cameras (including but not limited to area array cameras, line scan cameras, etc.), industrial lenses (including but not limited to long-focus lenses, short-focus lenses, telecentric lenses, zoom lenses, etc.) and visual light sources (including but not limited to bar light sources, ring light sources, coaxial light sources, etc.).

[0077] It should be noted that image preprocessing mainly includes two functions: image denoising and contrast enhancement. Image denoising includes but is not limited to Gaussian filtering, median filtering, mean filtering, bilateral filtering, etc.; contrast enhancement includes but is not limited to histogram equalization, histogram stretching, power transform, logarithmic transform, etc.; defect filtering can be achieved through point filters, line filters or cluster filters (see the patent "Defect Detection Method, Device, Defect Detection Equipment and Computer Storage Medium" for details); defect segmentation refers to the binarization processing of defect filtering results, and defect segmentation methods include but are not limited to threshold segmentation method, OTSU method, global threshold method, adaptive threshold method, etc.

[0078] Step S20, inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge;

[0079] In a feasible implementation, the step of inputting the defect candidate area into a traditional feature filter to obtain a traditional output result based on prior knowledge includes: extracting each connected domain of the defect candidate area; performing feature extraction on each connected domain based on a target feature value set to obtain a feature value set; determining the defect area and normal area in the defect candidate area based on the feature value set and the target feature value set; and determining the traditional output result based on the defect area and the normal area.

[0080] It should be noted that, for the segmented binary image B(x,y), each connected domain {C1, C2, C3,…, C n}, and extract features of each connected domain (including but not limited to the size, length, width, contrast, rectangularity, circularity, aspect ratio and other features of the region), let the connected domain C m The feature set is { F1, F2, F3, …, F k}, based on the known defect features (i.e., the target feature value set), feature screening is performed on each connected domain to finally obtain the defect area. Feature screening is performed according to the following principles:

[0081]

[0082] Where i=(1,2,3,…k), T1 is the minimum threshold set for each feature in the target feature value set, T2 is the maximum threshold set for each feature in the target feature value set, and C m The criteria for determining defective areas are:

[0083]

[0084] It should be noted that when D m When it is equal to 1, it means that area C m It is a defective area (that is, each filtered feature is within the feature range set for each feature in the target feature value set), otherwise it is a normal area.

[0085] Step S30: In the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain the artificial intelligence output result, and the defect judgment result is determined according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0086] It's important to note that mutual supervision refers to combining traditional and AI outputs to determine defects. Entering mutual supervision mode indicates that the deep feature extraction network is a successfully trained model, capable of accurately extracting features from defect candidate areas.

[0087] In a feasible embodiment, in the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, the defect judgment result is determined according to the mutual supervision strategy. The steps include: determining the judgment preference value of the target feature value set; if the judgment preference value is a first value, then entering the non-mutual supervision detection mode, using the traditional output result as the defect judgment result, and training the deep feature extraction network with the traditional initial result as the defect sample; if the judgment preference value is a second value, then entering the mutual supervision detection mode, inputting the defect candidate area into the deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determining the defect judgment result according to the mutual supervision strategy.

[0088] It should be noted that since deep learning networks require a large number of training samples, when they are initially running or encounter new types of defects, it is impossible to provide a large number of training samples to train the model. This will make the deep learning network unusable in this case. Therefore, to address these problems, a detection strategy based on judgment preference values is set up, as follows:

[0089] During the feature screening process, a judgment preference setting is added to each set of target feature value sets used for feature screening (the judgment preference value can be used to determine which detection mode to enter. For example, when the judgment preference value is 1, the non-mutually supervised detection mode is entered, and when the judgment preference value is 0, the mutual supervision detection mode is entered). This setting is used to distinguish whether the target feature value sets are independent for defect judgment, thereby solving the problem of insufficient system samples.

[0090] The judgment preference value is defined as: State = {1, 0}

[0091] 1. When State = 1 (that is, the preference judgment value is the first value), only the defect areas screened by the set of target feature values are output, the defect candidate areas are not sent to the AI network for judgment, and the screened defect samples are saved. These defect samples can be used to train the deep learning model (that is, the deep feature extraction network), so that a certain degree of defect detection can still be maintained even if there are not enough defect samples (traditional feature screening is used for defect detection).

[0092] 2. When State = 0 (i.e., the preference judgment value is the second value), the mutual supervision detection mode is entered, and the output needs to combine the traditional image algorithm (i.e., traditional feature filter) and the AI deep network (deep feature extraction network) to jointly determine the defect judgment result.

[0093] In a feasible implementation, the step of determining the defect judgment result based on the traditional output result and the artificial intelligence output result according to the mutual supervision strategy includes: judging whether the traditional output result is the same as the artificial intelligence output result; when it is judged that the traditional output result is the same as the artificial intelligence output result, taking the same result as the defect judgment result; when it is judged that the traditional output result is different from the artificial intelligence output result, determining the feature confidence of the target feature value set, and determining the defect judgment result based on the feature confidence.

[0094] It should be noted that when the traditional output result is the same as the artificial intelligence output result, the same result can be directly used as the defect judgment result; when the traditional output result is different from the artificial intelligence output result, it is necessary to further determine the defect judgment result based on the feature confidence.

[0095] In a feasible embodiment, the step of determining the defect judgment result based on the feature confidence includes: when the feature confidence is a first value and the traditional output result is a defect area, taking the traditional output result as the defect judgment result, and retraining the deep feature extraction network with the traditional output result as a defect sample; when the feature confidence is a first value and the traditional output result is a normal area, taking the artificial intelligence output result as the defect judgment result, and updating the screening threshold of the target feature value set; when the feature confidence is a second value and the traditional output result is a defect area, taking the artificial intelligence output result as the defect judgment result; when the feature confidence is a second value and the traditional output result is a normal area, taking the traditional output result as the defect judgment result, and retraining the deep feature extraction network with the traditional output result as a defect sample.

[0096] It should be noted that the first value is set to 1 and the second value is set to 0. Figure 2 As shown in the figure, when the traditional output result is the same as the artificial intelligence output result, the same result is directly used as the defect judgment result; when the traditional output result is different from the artificial intelligence output result, it is necessary to further determine the feature confidence of the target feature value set.

[0097] In a specific implementation, when the feature confidence is 1 and the traditional output result determines that the defect candidate area is a defect area, this indicates that the artificial intelligence output result determines that the defect candidate area is a normal area (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence is 1, it is very certain that the defect candidate area is a defect area. Therefore, the traditional output result can be used as the defect determination result. Since the artificial intelligence output result is inaccurate (indicating AI failure), the traditional output result can be saved as a defect sample and used for later updating the AI model (i.e., deep feature extraction network); when the feature confidence is 1 and the traditional output result determines that the defect candidate area is a normal area, this indicates that the artificial intelligence output result determines that the defect candidate area is a defect area (because the traditional output result is inconsistent with the artificial intelligence output result). Since the condition for defect feature screening is more stringent when the feature confidence is 1, the artificial intelligence output result can be used as the defect determination result. Since the traditional output result determines that the defect candidate area is a normal area, it can be determined that the traditional feature filter used to output the traditional output result has failed. At this time, it is necessary to update the screening threshold of the target feature value set so that the traditional feature filter can ensure that the traditional output result is a defect area when the feature confidence is 1.

[0098] In the specific implementation, when the feature confidence is 0 and the traditional output result determines that the defect candidate area is a defect area, this means that the artificial intelligence output result determines that the defect candidate area is a normal area (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence is 0, it is uncertain whether the defect candidate area is a defect area, so the artificial intelligence output result can be directly used as the defect judgment result; when the feature confidence is 0 and the traditional output result determines that the defect candidate area is a normal area, this means that the artificial intelligence output result determines that the defect candidate area is a defect area (because the traditional output result is inconsistent with the artificial intelligence output result). Since the feature confidence is 0, the conditions for defect feature screening are relatively loose, so the system is more inclined to believe that the candidate area is a normal area. Therefore, the traditional output result can be used as the defect judgment result. Since the artificial intelligence output result is inaccurate (indicating that AI fails), the traditional output result can be saved as a defect sample and used for later updating the AI model (i.e., deep feature extraction network).

[0099] It should be noted that in the process of feature screening, a feature confidence is added to each set of target feature value sets used for feature screening to distinguish the degree of credibility of the set of feature sets for defect judgment, so as to cooperate with the AI discrimination system for mutual supervision detection.

[0100] Define confidence as: Trust = {1, 0}

[0101] 1. When Trust = 1 (that is, the feature confidence is the first value), it means that the area filtered by the set of target feature values is very certain that the traditional algorithm is a defect area. At this time, it tends to trust the results of the traditional feature filter.

[0102] 2. When Trust = 0 (i.e., the feature confidence is the second value), it means that the algorithm is not sure whether the area selected by the set of target feature values is a defect area. At this time, it tends to trust the output results of the deep feature extraction network.

[0103] This embodiment determines a defect candidate area based on the target product image; inputs the defect candidate area into a traditional feature filter to obtain a traditional output result based on prior knowledge; in a mutual supervision detection mode, inputs the defect candidate area into a deep feature extraction network to obtain an artificial intelligence output result, and determines a defect judgment result based on the traditional output result and the artificial intelligence output result according to a mutual supervision strategy, thereby solving the technical problem that there are deficiencies in using a single traditional image algorithm or deep learning technology for product defect detection. Compared with the existing technology, the present application can combine traditional feature filters and deep feature extraction networks to jointly perform defect judgment on target product images, and improve the defect recognition capabilities of traditional feature filters and deep feature extraction networks through mutual supervision learning, thereby achieving accurate defect judgment.

[0104] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , step S30 also includes steps S301 to S303:

[0105] Step S301, inputting the defect candidate region into the deep feature extraction network to obtain feature information of the defect candidate region;

[0106] It should be noted that the deep feature extraction network can be used to perform accurate feature extraction on defect candidate areas.

[0107] Step S302, determining the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database, and using the maximum cosine similarity as the output score;

[0108] It should be noted that the feature database pre-stores some feature vectors of defect areas (i.e., defect features). After the deep feature extraction network extracts the corresponding feature information of the defect candidate area from the defect candidate area, the cosine similarity is calculated one by one with the feature vectors stored in the feature database, and the maximum cosine similarity is used as the output score.

[0109] Step S303: Determine an artificial intelligence output result based on a comparison result between the output score and the matching score threshold.

[0110] It should be noted that the artificial intelligence output result can be determined based on the comparison result between the output score and the matching score threshold. For example, when the output score is greater than or equal to the matching score threshold, it means that the artificial output result determines that there is a defective area in the defect candidate area. When the output score is less than the matching score threshold, it means that the artificial output result determines that the defect candidate area is a normal area.

[0111] In a feasible embodiment, the deep feature extraction network includes a shape weight correction module and a shape attention module, the shape weight correction module includes an SW coefficient layer, a maximum pooling layer, a mean pooling layer and a splicing layer, and the shape attention module includes a channel attention layer and a defect shape weight attention layer; wherein, the step of inputting the defect candidate area into the deep feature extraction network to obtain feature information of the defect candidate area includes: inputting the defect candidate area into the shape weight correction module, determining the first extraction feature through the SW coefficient layer and the maximum pooling layer, determining the second extraction feature through the mean pooling layer, and splicing the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer; inputting the third extraction feature into the shape attention module, and determining the feature information of the defect candidate area through the channel attention layer and the defect shape weight attention layer.

[0112] It should be noted that the deep feature extraction network is a more efficient deep neural network structure designed for defect shapes. It is used to accurately extract the characteristic expression of defects. In this network structure, we have designed two new plug-and-play modules:

[0113] 1. Shape Weight Correction Module (SWCM). SWCM replaces the single pooling operation commonly used in network structures. It can solve the problem that convolutional networks lack a bias in learning feature weight coefficients for target defects and background areas during global feature extraction.

[0114] 2. Shape Block Attention Module (SBAM). SBAM introduces a defect shape attention module (i.e., defect shape attention layer) based on the Convolutional Block Attention Module (CBAM) to improve the model's ability to extract defect shape information.

[0115] It should be noted that if Figure 4 As shown, the new feature X1 (i.e., the first extracted feature) of the SW coefficient correction pool is Figure 5 The new feature X2 (i.e., the second extracted feature) obtained by the mean pooling shown in the figure is concatenated to obtain the following Figure 6 The new output feature X3 (i.e., the third extracted feature) shown in FIG3 retains both the global information of the input feature and the corrected detail information, thereby enriching the expression capability of the output feature of the convolution module.

[0116] It should be noted that if Figure 7 As shown in the figure, the shape attention module is based on the design idea of CBAM and proposes the SBAM structure. On the basis of the CBAM structure, the channel attention layer (CAM, Channel Attention Module) is retained and the spatial attention layer (SAM, Spatial Attention Module) is replaced by the defect shape weight attention layer (SWAM, ShapeWeight Attention Module).

[0117] It should be noted that if Figure 8 As shown, for the SWAM layer:

[0118] a) For the input feature map, use the SW coefficient to correct the weight of each channel;

[0119] b) Perform maximum pooling and mean pooling operations on the corrected new feature map;

[0120] c) Concatenate the maximum pooling and mean pooling features, then connect a convolutional layer to extract features to obtain new features with channel compression of 1 and shape and spatial information preserved;

[0121] d) Perform a Sigmoid activation on the new feature to obtain the final shape weighted attention feature;

[0122] e) The SW coefficients in the SWAM module and the SW coefficients used in the SWCM module are reusable, so there is no additional cost. This is because for an image, the SW coefficients only need to be generated once and can be used by different modules in the network, so there is no need to generate SW coefficients multiple times for different modules;

[0123] In the specific implementation, Figure 9 As shown in Figure 1, the defect candidate area passes through the SWCM structure and the SBAM structure in turn to obtain the network output (i.e., the feature information of the defect candidate area).

[0124] In a feasible embodiment, the step of determining the first extraction feature through the SW coefficient layer and the maximum pooling layer includes: obtaining the original defect image; determining the image mean of the original defect image according to the saliency algorithm; determining the SW coefficient of the SW coefficient layer based on the original defect image, the image mean and the stretching coefficient; processing the defect candidate area through the SW coefficient of the SW coefficient layer to obtain the initial extraction feature, and obtaining the first extraction feature after processing the initial extraction feature through the maximum pooling layer.

[0125] It should be noted that there are two ways to construct the SW coefficients in the SW coefficient layer. One is to automatically calculate the coefficients by the algorithm based on the significant features, and the other is to generate them by manually annotating the data.

[0126] In the specific implementation, Figure 10 As shown, the algorithm automatically calculates the coefficient of significant features: the saliency algorithm is used to automatically calculate the weight of the significant target in the whole image according to the input image:

[0127]

[0128] Among them, I u represents the image mean calculated by the saliency algorithm, Iwh(x,y) represents the original image (i.e., the original defect image), α represents the stretching coefficient of the image mean, and I w The saliency algorithm calculates the distance between each pixel value of the original image and the image mean, and takes the absolute value to obtain the coefficient map as the SW coefficient.

[0129] In the specific implementation, it is generated by manually annotated data: while manually annotating the defect image category, the weight coefficient of the defect area is manually annotated, for example: the defect area is marked as 1 and the background is marked as 0, and a new binary image is generated as the manual SW coefficient.

[0130] It should be noted that after the SW coefficient is dot-multiplied with the input feature (the convolutional layer feature of the defect candidate area) to obtain a new feature (i.e., the initial extracted feature), a new pooled feature X1 (i.e., the first extracted feature) is obtained after a maximum pooling operation. Since the initial extracted feature is enhanced by the shape weight coefficient, the feature weight of the defect foreground area is enhanced, while the feature weight of the background area is suppressed.

[0131] It should be noted that the SW coefficient is determined during the network structure design and data annotation, so no additional coefficient generation consumption will be generated during the inference process.

[0132] This embodiment inputs the defect candidate region into the deep feature extraction network to obtain feature information of the defect candidate region; determines the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database, and uses the maximum cosine similarity as the output score; and determines the artificial intelligence output result based on the comparison result of the output score and the matching score threshold. In this way, the deep feature extraction network can accurately extract the characteristic expression of the defect, thereby obtaining a more accurate artificial intelligence output result.

[0133] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the mutual supervision defect detection method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0134] This application also provides a mutual supervision defect detection device, please refer to Figure 11 , the mutual supervision defect detection device includes:

[0135] A determination module 10 is configured to determine a candidate defect area based on the target product image;

[0136] An input module 20, configured to input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge;

[0137] The determination module 10 is also used to input the defect candidate area into the deep feature extraction network in the mutual supervision detection mode to obtain the artificial intelligence output result, and determine the defect judgment result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result.

[0138] The mutual supervision defect detection device provided in this application, which employs the mutual supervision defect detection method of the aforementioned embodiment, can resolve the technical issues inherent in product defect detection using either a single traditional image algorithm or deep learning technology. Compared to the prior art, the beneficial effects of the mutual supervision defect detection device provided in this application are the same as those of the mutual supervision defect detection method of the aforementioned embodiment, and the other technical features of the mutual supervision defect detection device are the same as those disclosed in the aforementioned embodiment, and are not further elaborated here.

[0139] The present application provides a mutual supervision defect detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the mutual supervision defect detection method in the above-mentioned embodiment one.

[0140] Reference below Figure 12, which shows a schematic diagram of the structure of a mutual supervision defect detection device suitable for implementing the embodiments of the present application. The mutual supervision defect detection device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12 The mutual supervision defect detection device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 12 As shown, the mutual supervision defect detection device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the mutual supervision defect detection device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the mutual supervision defect detection device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a mutual supervision defect detection device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0142] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The mutual supervision defect detection device provided by this application, which uses the mutual supervision defect detection method in the above-mentioned embodiment, can solve the technical problem that the use of a single traditional image algorithm or deep learning technology for product defect detection is insufficient. Compared with the existing technology, the beneficial effects of the mutual supervision defect detection device provided by this application are the same as the beneficial effects of the mutual supervision defect detection method provided by the above-mentioned embodiment, and the other technical features of the mutual supervision defect detection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0146] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the mutual supervision defect detection method in the above-mentioned embodiment.

[0147] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0148] The computer-readable storage medium may be included in the mutual-supervision defect detection device; or may exist independently without being assembled into the mutual-supervision defect detection device.

[0149] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the mutual supervision defect detection device, the mutual supervision defect detection device enables the following: to determine the defect candidate area based on the target product image; to input the defect candidate area into the traditional feature filter to obtain the traditional output result based on prior knowledge; in the mutual supervision detection mode, to input the defect candidate area into the deep feature extraction network to obtain the artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, determine the defect judgment result according to the mutual supervision strategy.

[0150] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0153] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned mutual supervision defect detection method. This computer-readable storage medium can address the technical issues inherent in product defect detection using either traditional image algorithms or deep learning techniques. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the mutual supervision defect detection method provided in the aforementioned embodiments and are not further elaborated upon here.

[0154] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned mutual supervision defect detection method when executed by a processor.

[0155] The computer program product provided in this application can address the technical issues inherent in product defect detection using either traditional image algorithms or deep learning techniques. Compared to existing technologies, the beneficial effects of the computer program product provided in this application are similar to those of the mutual supervision defect detection method provided in the aforementioned embodiments, and are not further elaborated here.

[0156] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A mutual supervision defect detection method, characterized in that: The method comprises: Determine defect candidate areas based on the target product image; Inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; In the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain an artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, the defect judgment result is determined according to the mutual supervision strategy; The step of determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result includes: Determining whether the traditional output result is the same as the artificial intelligence output result; When it is determined that the traditional output result is different from the artificial intelligence output result, determining a feature confidence of the target feature value set, and determining a defect determination result based on the feature confidence; The step of determining the defect determination result according to the feature confidence level includes: When the feature confidence is a first value and the traditional output result is a defect area, using the traditional output result as a defect determination result, and retraining the deep feature extraction network after using the traditional output result as a defect sample; When the feature confidence is a first value and the traditional output result is in a normal area, taking the artificial intelligence output result as a defect determination result, and updating the screening threshold of the target feature value set; When the feature confidence is a second value and the traditional output result is a defect area, using the artificial intelligence output result as a defect determination result; When the feature confidence is the second value and the traditional output result is a normal area, the traditional output result is used as a defect determination result, and the deep feature extraction network is retrained after the traditional output result is used as a defect sample.

2. The method according to claim 1, wherein The step of inputting the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge includes: Extracting connected domains of the defect candidate area; Based on the target eigenvalue set, features are extracted from each connected domain to obtain a eigenvalue set; Determining a defect area and a normal area in the defect candidate area based on the feature value set and the target feature value set; A conventional output result is determined according to the defective area and the normal area.

3. The method according to claim 2, wherein The steps of inputting the defect candidate area into the deep feature extraction network in the mutual supervision detection mode to obtain an artificial intelligence output result, and determining the defect determination result according to the mutual supervision strategy based on the traditional output result and the artificial intelligence output result include: Determining a judgment preference value of the target feature value set; If the determination preference value is the first value, then after entering the non-mutually supervised detection mode, the traditional output result is used as the defect determination result, and the traditional initial result is used as the defect sample to train the deep feature extraction network; If the judgment preference value is the second value, after entering the mutual supervision detection mode, the defect candidate area is input into the deep feature extraction network to obtain the artificial intelligence output result, and based on the traditional output result and the artificial intelligence output result, the defect judgment result is determined according to the mutual supervision strategy.

4. The method according to claim 1 or 3, wherein: The step of inputting the defect candidate area into the deep feature extraction network to obtain an artificial intelligence output result includes: Inputting the defect candidate region into the deep feature extraction network to obtain feature information of the defect candidate region; Determine the maximum cosine similarity between the defect candidate region feature information and the defect features stored in the feature database, and use the maximum cosine similarity as the output score; An artificial intelligence output result is determined based on a comparison result of the output score and a matching score threshold.

5. The method according to claim 4, wherein The deep feature extraction network includes a shape weight correction module and a shape attention module, the shape weight correction module includes an SW coefficient layer, a maximum pooling layer, a mean pooling layer and a splicing layer, and the shape attention module includes a channel attention layer and a defect shape weight attention layer; wherein, the step of inputting the defect candidate area into the deep feature extraction network to obtain feature information of the defect candidate area includes: Input the defect candidate area into the shape weight correction module, determine the first extraction feature through the SW coefficient layer and the maximum pooling layer, determine the second extraction feature through the mean pooling layer, and splice the first extraction feature and the second extraction feature into a third extraction feature through the splicing layer; The third extracted feature is input into the shape attention module, and the defect candidate area feature information is determined through the channel attention layer and the defect shape weight attention layer.

6. The method according to claim 5, wherein The step of determining the first extracted feature through the SW coefficient layer and the maximum pooling layer includes: Obtaining the original defect image; Determining an image mean of the original defect image according to a saliency algorithm; determining a SW coefficient of the SW coefficient layer based on the original defect image, the image mean, and a stretching coefficient; The defect candidate area is processed by the SW coefficient of the SW coefficient layer to obtain an initial extraction feature, and the initial extraction feature is processed by the maximum pooling layer to obtain a first extraction feature.

7. The method according to claim 1, wherein After the step of determining whether the traditional output result is the same as the artificial intelligence output result, the method further includes: When it is determined that the traditional output result is the same as the artificial intelligence output result, the same result is used as the defect determination result.

8. The method according to claim 1, wherein The step of determining the defect candidate area according to the target product image includes: Performing image preprocessing on the target product image to obtain a preprocessed product image, wherein the image preprocessing includes image denoising and contrast enhancement; performing defect filtering on the preprocessed product image to obtain a defect filtered image; Binarization is performed on the defect filtered image to obtain a defect candidate area.

9. A mutual supervision defect detection device, characterized in that: The device comprises: A determination module, configured to determine a defect candidate area based on a target product image; An input module, configured to input the defect candidate region into a traditional feature filter to obtain a traditional output result based on prior knowledge; The determination module is further configured to input the defect candidate region into a deep feature extraction network in a mutual supervision detection mode to obtain an artificial intelligence output result, and determine a defect determination result according to a mutual supervision strategy based on the traditional output result and the artificial intelligence output result; The method of determining a defect determination result based on the traditional output result and the artificial intelligence output result according to a mutual supervision strategy includes: Determining whether the traditional output result is the same as the artificial intelligence output result; When it is determined that the traditional output result is different from the artificial intelligence output result, determining a feature confidence of the target feature value set, and determining a defect determination result based on the feature confidence; Determining the defect determination result according to the feature confidence level includes: When the feature confidence is a first value and the traditional output result is a defect area, using the traditional output result as a defect determination result, and retraining the deep feature extraction network after using the traditional output result as a defect sample; When the feature confidence is a first value and the traditional output result is in a normal area, taking the artificial intelligence output result as a defect determination result, and updating the screening threshold of the target feature value set; When the feature confidence is a second value and the traditional output result is a defect area, using the artificial intelligence output result as a defect determination result; When the feature confidence is the second value and the traditional output result is a normal area, the traditional output result is used as a defect determination result, and the deep feature extraction network is retrained after the traditional output result is used as a defect sample.

Images