Defect detection methods, apparatus, electronic equipment and computer-readable storage media

Abstract

This application relates to the field of industrial automation technology, and provides a defect detection method, device, electronic device, and computer-readable storage medium. The method includes: acquiring an image to be detected; inputting the image to be detected into a pre-stored attention-based generative adversarial network (GAN) for processing, and outputting a target image; and determining the defect region in the target image. Because the attention-based GAN introduces an attention mechanism, the foreground and background images under the attention of the attention mechanism are clearly distinguished. By processing the image to be detected using the attention-based GAN, the obtained target image can be an image that enhances, modifies, or removes the foreground image. Using such a target image to detect defect regions enables automatic detection of defects in the object to be detected in the image, eliminating the need for manual annotation, improving detection efficiency, and saving labor costs.

Description

Technical Field

[0001] This application relates to the field of industrial automation technology, and in particular to a defect detection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In industrial production, product quality inspection is an extremely important process. The most crucial step in quality inspection is defect detection. To help locate the source of defects in industrial production, it is typically necessary to detect, classify, and segment the defective regions in images of the object being inspected.

[0003] Currently, methods for detecting, classifying, and segmenting industrial defect areas in images of inspected objects require detailed contour annotation of the images containing industrial defects. This manual annotation process is labor-intensive and time-consuming. Therefore, an automated defect detection method is needed to improve the efficiency of screening defective images and save on quality inspection labor costs. Summary of the Invention

[0004] This application provides a defect detection method, apparatus, electronic device, and computer-readable storage medium, which can solve the problem of automatic defect detection.

[0005] In a first aspect, embodiments of this application provide a defect detection method, including:

[0006] Acquire the image to be detected;

[0007] The image to be detected is input into a pre-stored attention-based generative adversarial network for processing, and the target image is output.

[0008] Identify the defect areas in the target image.

[0009] It should be understood that, due to the introduction of the attention mechanism in the attention generative adversarial network, the foreground and background images that are focused on by the attention mechanism are clearly distinguished. The image to be detected is input into the pre-stored attention generative adversarial network for processing, and the target image obtained can be an image that enhances, modifies, or removes the foreground image. Using such a target image to detect defect regions, the defects of the object to be detected in the image can be automatically detected, which can save the manual annotation step, improve detection efficiency, and save labor costs.

[0010] Secondly, embodiments of this application provide a defect detection device, including:

[0011] The image acquisition module is used to acquire the image to be detected.

[0012] The image processing module is used to input the image to be detected into a pre-stored attention-based generative adversarial network for processing and output the target image;

[0013] The defect region determination module is used to determine the defect region in the target image.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method steps described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method steps described in the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method steps described in the first aspect.

[0017] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an attention-based generative adversarial network structure provided in an embodiment of this application;

[0020] Figure 2 This is a schematic flowchart of a defect detection method provided in an embodiment of this application;

[0021] Figure 3A1 The image to be detected is an input to an attention-generating adversarial network provided in one embodiment of this application;

[0022] Figure 3A2 This is the background image output by an attention-generating adversarial network provided in an embodiment of this application;

[0023] Figure 3A3 This is an attention mask image output by an attention generation adversarial network provided in an embodiment of this application;

[0024] Figure 3B1 The image to be detected is an input to an attention-generating adversarial network provided in another embodiment of this application;

[0025] Figure 3B2 This is the background image output by the attention-generating adversarial network provided in another embodiment of this application;

[0026] Figure 3B3 This is an attention mask image output by an attention generation adversarial network provided in another embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating a defect detection method provided in another embodiment of this application;

[0028] Figure 5 This is a flowchart illustrating a defect detection method provided in another embodiment of this application;

[0029] Figure 6A1 The image to be detected is an input to an attention-generating adversarial network provided in one embodiment of this application;

[0030] Figure 6A2 This is an attention mask image output by an attention generation adversarial network provided in an embodiment of this application;

[0031] Figure 6A3 This is an example of using defect boxes to annotate defect areas in an image to be detected, provided in one embodiment of this application;

[0032] Figure 6B1 The image to be detected is an input to an attention-generating adversarial network provided in another embodiment of this application;

[0033] Figure 6B2 This is an attention mask image output by an attention generation adversarial network provided in an embodiment of this application;

[0034] Figure 6B3 This is an example diagram of a defect region in an image to be detected, provided by an embodiment of this application;

[0035] Figure 7 This is a flowchart illustrating a defect detection method provided in another embodiment of this application;

[0036] Figure 8A This is an example diagram of defect region segmentation in a target image provided in an embodiment of this application;

[0037] Figure 8B This is an example diagram of defect region segmentation in a target image provided in another embodiment of this application;

[0038] Figure 9 This is a schematic diagram of the defect detection device provided in the embodiments of this application;

[0039] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0042] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0043] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0044] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] Before describing the defect detection method provided in the embodiments of this application, the principle of the defect detection method provided in the embodiments of this application and the related concepts involved in the embodiments of this application will be explained below to facilitate understanding of the embodiments of this application.

[0047] In industrial production, product quality inspection is an extremely important process. The most crucial part of quality inspection is defect detection.

[0048] In some industrial settings, during product manufacturing, automated quality inspection equipment based on industrial vision acquires images of the objects to be inspected and performs initial screening. This screening identifies a large number of images containing industrial defects. To help pinpoint the source of these defects in industrial production, it is typically necessary to detect, classify, and segment the defective regions within these images. The detection and segmentation of industrial defect images requires detailed contour annotation on a large number of these images, consuming significant manpower for manual annotation and resulting in a lengthy annotation process.

[0049] In other scenarios, current deep learning-based defect classification methods require labeling defects in training images, which is inefficient through manual labeling. To be applicable to various types of defects, a certain number of defect images need to be collected as training samples for each category. However, the distribution of defect types on actual production lines is uneven, and it is difficult to collect enough images of some less common but severely impactful defects for training the segmentation network, thus slowing down the development iteration cycle.

[0050] For factories, there is a greater need for universal defect / detection segmentation algorithms that can handle different shooting parameters and different types of defects, and can efficiently process defect images screened by the machine, thereby saving on quality inspection manpower costs.

[0051] Therefore, a method is needed to automatically detect defects. On the one hand, it can improve the efficiency of screening defective images; on the other hand, it can provide training samples for deep learning-based defect classification models, or automatically label the defects of the detected objects when there are few training samples; and on yet another hand, it can be a general defect / detection segmentation algorithm that can handle different shooting parameters and different defect types.

[0052] Among various defect detection and segmentation methods, single-image-based segmentation algorithms are relatively mature, mainly categorized into clustering, edge detection, region growing, graph segmentation, and deep learning networks. For industrial products, most severe defects occur on specific components, and the texture features of these defects are affected by the component's texture, impacting the accuracy of clustering or region growing-based segmentation. Furthermore, the illumination in defect images captured by automated quality inspection equipment is inconsistent, and cameras are prone to defocusing during movement, resulting in some defects having low contrast and unclear edges. Inconsistent image brightness and unclear edges limit the performance of segmentation methods such as edge detection, region growing, and graph segmentation. Therefore, the industry currently also utilizes deep learning methods for defect segmentation within industrial visual images, such as Mask R-CNN, Unet, and DeepLab, among other deep learning networks used for image segmentation. The main challenge of deep learning-based defect segmentation methods lies in the need to annotate defects in training images at the pixel level, which results in low annotation efficiency. In order to be applicable to various types of defects, a certain number of defect images need to be collected for training for each type. However, the distribution of defect types on actual production lines is uneven, and it is difficult to collect enough images of some less common defects that have a serious impact on circuits for training the segmentation network, which slows down the development and iteration cycle.

[0053] This application provides a defect detection method based on Attention Generative Adversarial Networks (AttentionGAN), which can automatically label and segment defects in industrial images. The overall accuracy of automatic labeling and detection is much higher than that of traditional algorithms, and it is more efficient and less costly than manual labeling, with a wide range of applications.

[0054] Attention-based generative adversarial networks (GANs) introduce an attention network into recurrent generative adversarial networks (CycleGANs). Figure 1 This is a schematic diagram of the structure of an attention generative adversarial network provided in an embodiment of this application. Figure 1 The diagram shows an input image x from the source domain X, which is processed by an attention network to generate an image G from the target domain Y. y The process, and the generated image G y The process of reconstructing the image Rx from the source domain X using an attention network.

[0055] In the process of training the attention generative adversarial network provided in the embodiments of this application, the attention mask generator from the source domain X to the target domain Y is trained primarily through asymmetric training samples. Content Mask Generator (from source domain X to target domain Y) After the network model is trained, during inference, the attention mask generator processes the input image and generates a foreground attention mask. and background attention mask The content mask generator can generate the content mask to be transformed from the attention mask portion of a multi-channel application. y .

[0056] As shown in formula (1) below, the process of generating the image includes a foreground attention mask. and background attention mask Its function.

[0057]

[0058] Figure 1 In the attention generative adversarial network shown, the attention mask generator produces a 1-channel background attention mask. Foreground attention mask for n-1 channels Foreground attention mask This highlights the foreground (horse) of the input image x, while ignoring the background. This is the foreground attention mask. With content mask C y Performing dotwise multiplication will highlight the horse, while the background will be placed at a very low value. This background attention mask... The foreground of the input image x is multiplied pointwise by the input image x to obtain the background of the input image x, while the foreground of the input image x is set to a very low value. Then, the two are added pointwise to obtain the final output image with the foreground removed, i.e., the generated image G. y The number of channels, n, can be chosen empirically or determined through multiple tests. The advantage of this approach is that the image conversion primarily focuses on the foreground portion requiring conversion, while the background essentially does not need to be transformed. Figure 1 symbols in Indicates addition by dots, symbol Indicates dot-wise multiplication, symbol This indicates that the data is categorized by channel.

[0059] It should be understood that since the reconstruction process from the target domain Y to the source domain X in CycleGAN is consistent with the above process, the reconstruction process when training the attention-based recurrent adversarial network, that is, the generation process from the target domain Y to the source domain X, will not be elaborated here. For details, please refer to the introduction of attention-based generative adversarial networks in the existing technology.

[0060] It should be understood that Figure 1The attention-based generative adversarial network shown is merely an example; when implementing the embodiments of this application, it can be modified... Figure 1 The network can be adjusted to suit actual needs. In the embodiments of this application, attention generative adversarial networks that can obtain attention masks of the foreground image and / or obtain target domain output results of the foreground image can be applied to the embodiments of this application.

[0061] Figure 2 This application illustrates a defect detection method provided in an embodiment, applied to an electronic device. In a specific example, the electronic device can be a defect detection device, which can be implemented by the electronic device's software and / or hardware. Figure 2 As shown, the method includes steps S110 to S130. The specific implementation principle of each step is as follows:

[0062] S110, acquire the image to be detected.

[0063] In some embodiments of this application, the image to be inspected can be an image of the object being inspected, acquired directly on the production line by an electronic device through an image acquisition device, such as a camera. The object being inspected can be a device or product, such as a circuit board, that requires defect inspection.

[0064] In some other embodiments of this application, the image to be inspected can be an image acquired by an inspection machine on a production line. Alternatively, the image may have been pre-screened by the inspection machine, which initially determines that the object to be inspected contains industrial defects.

[0065] S120, the image to be detected is input into a pre-stored attention generative adversarial network for processing, and the target image is output.

[0066] The target image can be Figure 1 The image shown is the attention mask image of the intermediate output of the attention generative adversarial network; it can also be the final output image of the target domain, i.e., the generated image G. y The attention mask image can be a foreground attention mask for any channel. It can also be a background attention mask. In some embodiments, the attention generative adversarial network used only generates a foreground attention mask A. y Then the attention mask image can be the foreground attention mask A. y It can also be used as a background attention mask (1-A) y In some embodiments, the final output image of the target domain is a background image from which defects of the detected object are removed by an attention-generating adversarial network.

[0067] Figure 3A1 and Figure 3B1 This is an example of the image to be detected of the object being detected; Figure 3A2 and Figure 3B2 This is an example of a background image obtained by processing an attention-based generative adversarial network, after removing defects from the detected object. Figure 3A3 and Figure 3B3 This is an example of an attention mask image of the detected object obtained through attention generative adversarial network processing.

[0068] In some embodiments of this application, the electronic device uses a trained attention generative adversarial network to process the target image and extract the intermediate output attention mask image as the target image.

[0069] In other embodiments of this application, the electronic device uses a trained attention generative adversarial network to process the target image and extract the background image of the target domain after removing defects as the target image.

[0070] S130, Determine the defect region in the target image.

[0071] In some embodiments of this application, the target image is an attention mask image, which is an intermediate output of an attention generative adversarial network, and the attention mask image is subjected to image processing to determine defect regions in the target image.

[0072] In other embodiments of this application, the target image is the final output image of the target domain, and image processing is performed using the final output image of the target domain to determine defect regions in the target image. In one specific embodiment, a background image of the defect in the detected object may be removed using an attention-based generative adversarial network, and the defect region in the target image may be determined by comparing the difference between the background image and the input image.

[0073] In some embodiments, the image to be inspected is an image of a printed circuit board (PCB) to be inspected acquired by an inspection machine, which includes PCB inspection equipment in a PCB production line. The inspection machine may include image acquisition devices such as cameras, webcams, and depth cameras. The inspection machine is used to photograph the PCB to be inspected passing through its photographing area. The defect area corresponds to the location of an industrial defect in the PCB to be inspected. Industrial defects in the PCB to be inspected include, but are not limited to, warping, open circuits, short circuits, cracks, wrinkles, or copper foil peeling.

[0074] It should be understood that because attention-based generative adversarial networks (GANs) introduce an attention mechanism, they selectively process the foreground image, resulting in a clear distinction between the foreground and background images that receive attention. By processing the image to be detected using GANs, the resulting target image can be an image that enhances, modifies, or removes the foreground image. Therefore, the target image output by GANs can be used to automatically detect defects in the object being detected, eliminating the need for manual annotation, improving detection efficiency, and saving labor costs.

[0075] In the above Figure 2 Based on the embodiment of the defect detection method shown, Figure 4 This application illustrates one implementation of the defect detection method provided in the embodiments of the present application, such as... Figure 4 As shown, before acquiring the image to be detected in step S110, the method further includes step S001:

[0076] S001, an untrained attention generative adversarial network is trained using a source domain image set and a target domain image set to obtain an attention generative adversarial network; wherein, the image in the source domain image set is the first image, the image in the target domain image set is the second image, the first image is the image of the detected object with defects, the second image is the image of the detected object without defects, and the source domain image set and the target domain image set are unpaired image sets.

[0077] The source domain image is the image of the detected object containing defects, while the target domain image is the image of the detected object without defects.

[0078] In some embodiments of this application, the process of training an attention-based generative adversarial network (GAN) in an electronic device includes the following steps: First, the GAN learns to generate an image without industrial defects from a target image containing industrial defects, while retaining the original image content. Then, the GAN inputs the generated image without industrial defects into a discriminative model for discrimination, determining whether the model can identify whether the generated image is a synthetic image without industrial defects or a misclassified image without industrial defects, and calculates the discrimination loss. Simultaneously, another generative model of the GAN generates a reconstructed image containing industrial defects from the image of the target image without industrial defects; this reconstructed image with industrial defects is compared with the original input target image containing industrial defects, and a reconstruction loss is calculated. The discrimination loss and the reconstruction loss are combined to form the model training loss.

[0079] To achieve good processing results and avoid overfitting, deep learning requires a large amount of data for network training. However, large-scale industrial defect pairing datasets are currently very scarce because paired data is difficult to collect and subsequent alignment is challenging. To reduce the cost of collecting paired data, this application uses a non-paired dataset: a source domain image set X consisting of various images of objects containing defects to be detected, and a target domain image set Y consisting of images of objects without defects to be detected. Then, these source and target domain image sets are used to train an attention-based generative adversarial network.

[0080] In the above Figure 2 or Figure 4 Based on the embodiment of the defect detection method shown, Figure 5 This application illustrates one implementation of the defect detection method provided in the embodiments of the present application, such as... Figure 5 As shown, step S120, which inputs the image to be detected into a pre-stored attention generative adversarial network for processing and outputs the target image, can be replaced by step S120'; correspondingly, step S130 includes steps S1311 and S1312.

[0081] S120': Input the image to be detected into the pre-stored attention generative adversarial network for processing and output the attention mask image; use the attention mask image as the target image.

[0082] In some embodiments of this application, the electronic device utilizes, for example Figure 1 The attention-based generative adversarial network (GAN) shown extracts the attention mask image during the processing of the image to be processed, and uses the attention mask image as the target image. The attention mask image can be a foreground attention mask of any channel. It can also be a background attention mask. In other embodiments, the attention generative adversarial network used by the electronic device only generates a foreground attention mask A. y Then the attention mask image can be the foreground attention mask A. y It can also be used as a background attention mask (1-A) y ).

[0083] S1311, perform histogram matching on the attention mask image to obtain a balanced image.

[0084] Histogram matching, also known as histogram specification, is an image enhancement method that transforms the histogram of an image into a histogram of a specified shape.

[0085] It should be noted that the value range of the attention mask itself is relatively small. Histogram matching widens the value range, making it easier to determine the defect location through thresholding. Specifically, because the pixel grayscale values ​​of the defect area and the normal area are quite similar, generally around 50, it is not easy to set a fixed threshold to separate them. After the histogram matching operation, the grayscale range is widened, and the difference between the pixel grayscale values ​​of the defect area and the normal area becomes more obvious, making it easier to set a threshold to extract the area containing the defect of the detected object.

[0086] S1312, Determine the defect region in the equalization image.

[0087] In some embodiments of this application, determining defect regions in an equalized image includes:

[0088] Thresholding is applied to the equalized image to obtain the seed location;

[0089] Based on the seed location, a pre-stored region segmentation algorithm is used to determine the first region in the equalized image whose attention measure is greater than the first threshold. The first region is the defect region.

[0090] Thresholding can be understood as follows: an image includes a target object, a background, and noise. To extract the target object directly from a multi-valued digital image, a threshold T can be set to divide the image data into two parts: a group of pixels with a gray value greater than T and a group of pixels with a gray value less than T.

[0091] Among them, the region-based segmentation method is a segmentation technique based on directly finding regions. Region growth starts with a set of seed pixels representing different growth regions. Next, pixels in the neighborhood of the seed pixels that meet the conditions are merged into the growth region represented by the seed pixels, and the newly added pixels are used as new seed pixels to continue the merging process until no new pixels that meet the conditions can be found.

[0092] It should be noted that there is a transition region between the boundary region of the defect and the normal region. In some embodiments of this application, the central region of the defect is first obtained by thresholding, and then the true boundary of the defect is found by region growing, making the segmentation result of the defect region more accurate.

[0093] In some alternative embodiments, a defect bounding box is generated at the defect location in the equal attention mask image based on the boundary of the first region.

[0094] In other embodiments of this application, after acquiring the image to be detected, the method further includes:

[0095] Obtain the estimated defect length of the image to be detected;

[0096] Accordingly, determining the defect regions in the equalization image includes:

[0097] The parameters of the sliding window are determined based on the defect length;

[0098] The parameters of a sliding window are used to search for the region with the highest attention measure in the equalized image, and the region with the highest attention measure is the defect region.

[0099] The grayscale value of a pixel in an attention mask image is called the attention metric, which represents the degree to which the attention network pays attention to that pixel.

[0100] In some implementations, the image to be inspected acquired by the electronic device is an image pre-screened by the inspection machine. The inspection machine initially screens images of objects containing industrial defects. The inspection machine estimates the side length of the region using a neighborhood comparison method. The inspection machine roughly estimates the size of the defect on a low-resolution grayscale image and sends this size as known information along with the high-resolution image to the electronic device. The neighborhood comparison method compares the target pixel with its neighboring pixels; if the difference between the target pixel and its neighboring pixels is large, the pixel is considered to belong to the defect region. In a low-resolution image, one pixel can represent several high-resolution pixels. Therefore, if two adjacent pixel regions are found to have large differences in surrounding pixels in a low-resolution image, a sliding window with a side length of 2A may be used in the high-resolution image, where A is the coefficient of the resolution difference between the low-resolution and high-resolution images.

[0101] In some embodiments, adopt Figure 5 The defect detection method shown detects the defect area and marks the defect area of ​​the image to be detected with a box. The image to be detected with the marked defects can be used as a sample to train the defect classification network, thereby reducing the manual labeling work, improving efficiency and saving labor costs.

[0102] Figure 6A1 and Figure 6B1 This is an example of the image to be detected of the object being detected; Figure 6A2 and Figure 6B2 It is an attention mask image of the detected object obtained through attention generative adversarial network processing; Figure 6A3 and Figure 6B3 By applying attention mask images Figure 5 The defect area is determined by the defect detection method and an example is marked with a square defect box.

[0103] refer to Figure 1In typical applications of attention-based generative adversarial networks (GANs), when processing foreground images—for example, transforming a horse in a foreground image into a zebra, or transforming an expressionless human face in a foreground image into a smiling or other expressive face—the intermediate output attention mask (including the foreground attention mask) is crucial. and background attention mask The image used to help the generative adversarial network (GAN) highlight the foreground while ignoring the background. This ensures that only the horse is highlighted, while the background is rendered with very low values, thus avoiding over-processing and significant background modification. Therefore, the intermediate output attention mask image is used to obtain the generated image G. y Its auxiliary role.

[0104] The inventors of this application have creatively discovered that in the attention mask image output by a trained attention generative adversarial network, the attention metric (i.e., pixel grayscale value) of defective regions within the attention mask is significantly different from that of other normal regions. The inventors have creatively extracted this attention mask image for defect detection in industrial scenarios, enabling automatic defect detection of the tested object with a small number of defect training samples, thereby reducing labor costs.

[0105] In the above Figure 4 Based on the embodiment of the defect detection method shown, Figure 7 This application illustrates an implementation method for defect detection, as shown in the embodiments of this application. Figure 7 As shown, step S120, which inputs the image to be detected into a pre-stored attention generative adversarial network for processing and outputs the target image, can be replaced by step S120”; correspondingly, step S130 includes steps S1321 and S1322.

[0106] S120”, the image to be detected is input into a pre-stored attention generative adversarial network for processing, and a background image with defects removed from the image to be detected is output, and the background image is used as the target image.

[0107] In some embodiments of this application, the following are employed: Figure 4 The defect detection method shown employs an attention-based generative adversarial network trained on a training set. When an electronic device inputs the image to be detected into this network, its final output target domain image is the background image from which defects of the detected object have been removed. The electronic device then uses this background image as the target image for subsequent image post-processing.

[0108] S1321, compare the pixel similarity between the image to be detected and the target image.

[0109] In the embodiments of this application, the electronic device can compare the pixel similarity between the image to be detected and the target image using similarity measures based on single pixel comparisons, including but not limited to the sum of absolute difference (SAD) or the sum of squared errors (SSE); it can also use similarity measures based on pixel neighborhood comparisons, including but not limited to feature similarity (FSIM) or structural similarity (SSIM). Pixel similarity comparisons of images are existing technologies and will not be elaborated upon here.

[0110] S1322, the pixel block formed by pixels in the target image whose pixel similarity is lower than the similarity threshold is taken as the defect area.

[0111] In some embodiments of this application, the similarity threshold is a preset value. It can be understood that when the electronic device performs pixel-level comparison between the image to be detected and the target image, pixels with a similarity lower than the similarity threshold are highly likely to be pixels corresponding to the defect locations of the detected object. The electronic device marks pixel blocks composed of pixels with similarity lower than the similarity threshold as defect regions. It should be understood that the electronic device marking pixels in defect regions is equivalent to performing image segmentation on the image to be detected. Figure 8A ,and Figure 8B This shows an example of the segmentation results of industrial defects in the object being inspected in the image to be inspected.

[0112] In other embodiments of this application, after defining pixel blocks composed of pixels in the target image with pixel similarity below a similarity threshold as defect regions, the method further includes:

[0113] The target image is filtered using a pre-stored morphological filtering algorithm to obtain an optimized image that does not include interfering defect areas.

[0114] In a specific example, the electronic device uses an opening operation, i.e., erosion followed by dilation, to filter the image to be detected, which has been marked with defect regions. The purpose of morphological filtering is to remove irregularly shaped and small-sized regions. Due to image noise, comparing the pixel similarity between the image to be detected and the target image reveals regions with large differences, which may be caused by reconstruction errors. However, these regions are usually small in size, and morphological filtering can remove this noise to obtain more accurate segmentation results.

[0115] It should be understood and referenced. Figure 1In typical applications of attention-based generative adversarial networks (GANs), foreground images are processed, such as impersonating a horse as a zebra or an expressionless face as a smiling or other expressive face. The output image G in the target domain... y This refers to the final result required for using generative adversarial networks (GANs). In other words, the ultimate goal of applying attention-based GANs is to generate forged images. The inventors of this application have creatively discovered that, due to the attention mechanism, there is a significant difference between the foreground region of the input image to be detected and the foreground region of the output image in an attention-based GAN. If the input image to be detected contains industrial defects, and the output target domain image is a background image with the industrial defects removed, then by comparing the input and output images, the foreground image, i.e., the region of the industrial defects in the detected object, can be clearly highlighted. Due to the attention mechanism, the non-defect regions of the output background image are similar to those of the input image. Using the comparison results for the detection of industrial defect regions and for more refined image segmentation of industrial defect regions, automatic defect detection of the tested object can be achieved with a small number of defect training samples, thereby reducing labor costs.

[0116] In some embodiments, the purpose of defect segmentation is to obtain the defect location with pixel-level precision. The segmented output image can be further used to determine the intersection relationship between the defect and each component, and to evaluate the impact of the defect on the panel quality.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0118] Corresponding to the above Figure 2 The defect detection method shown Figure 9 The illustration shows a defect detection device 900 provided in an embodiment of this application, comprising:

[0119] The image acquisition module 910 is used to acquire the image to be detected.

[0120] The image processing module 920 is used to input the image to be detected into a pre-stored attention generative adversarial network for processing and output the target image.

[0121] The defect region determination module 930 is used to determine the defect region in the target image.

[0122] It is understood that the various implementation methods and combinations of implementation methods in the above embodiments and their beneficial effects are also applicable to this embodiment, and will not be repeated here.

[0123] Figure 10This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. In a specific example, the electronic device can be a defect detection device. Figure 10 As shown, the electronic device D10 of this embodiment includes: at least one processor D100 ( Figure 10 (Only one is shown) a processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on at least one processor D100. When the processor D100 executes the computer program D102, it implements the steps in any of the above method embodiments. Alternatively, when the processor D100 executes the computer program D102, it implements the functions of each module / unit in the above device embodiments.

[0124] Electronic device D10 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This electronic device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device D10 and does not constitute a limitation on electronic device D10. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, image acquisition devices, etc.

[0125] The processor D100 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0126] In some embodiments, memory D101 can be an internal storage unit of electronic device D10, such as a hard disk or memory of electronic device D10. In other embodiments, memory D101 can also be an external storage device of electronic device D10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device D100. Furthermore, memory D101 can include both internal storage units and external storage devices of electronic device D10. Memory D101 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory D101 can also be used to temporarily store data that has been output or will be output.

[0127] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0130] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A defect detection method characterized by, The method comprises: acquiring a to-be-detected image; inputting the to-be-detected image into a pre-stored attention generation adversarial network for processing, and outputting a target image; the attention generation adversarial network comprises an attention mask generator and a content mask generator, the attention mask generator is used for generating foreground attention masks and background attention masks, and the content mask generator is used for generating a content mask; the attention generation adversarial network is trained by a non-paired source domain image set and a target domain image set, the images in the source domain image set are images of detected objects containing defects, the images in the target domain image set are images of detected objects not containing defects, and the target image is an attention mask image output by the attention generation adversarial network or a background image in which defects in the to-be-detected image are removed; determining a defect region in the target image.

2. The method of claim 1, wherein, Before the acquiring of the to-be-detected image, the method further comprises: training an untrained attention generation adversarial network by using a source domain image set and a target domain image set to obtain the attention generation adversarial network; wherein the images in the source domain image set are first images, the images in the target domain image set are second images, the first images are images of detected objects containing defects, the second images are images of detected objects not containing defects, and the source domain image set and the target domain image set are non-paired image sets.

3. The method of claim 1 or 2, wherein, The inputting of the to-be-detected image into the pre-stored attention generation adversarial network for processing and the outputting of the target image comprise: inputting the to-be-detected image into the pre-stored attention generation adversarial network for processing, and outputting an attention mask image; The determining of the defect region in the target image comprises: performing histogram matching on the attention mask image to obtain an equalized image; determining a defect region in the equalized image.

4. The method of claim 3, wherein, The determining of the defect region in the equalized image comprises: performing thresholding processing on the equalized image to obtain a seed position; determining, according to the seed position, a first region in which an attention measure is greater than a first threshold in the equalized image by using a pre-stored region segmentation algorithm, and the first region is a defect region.

5. The method of claim 3, wherein, After the acquiring of the to-be-detected image, the method further comprises: acquiring an estimated defect length of the to-be-detected image; The determining of the defect region in the equalized image comprises: determining parameters of a sliding window according to the defect length; searching, by using the parameters of the sliding window, a region with the maximum attention measure in the equalized image, and the region with the maximum attention measure is a defect region.

6. The method of claim 2, wherein, The inputting of the to-be-detected image into the pre-stored attention generation adversarial network for processing and the outputting of the target image comprise: inputting the to-be-detected image into the pre-stored attention generation adversarial network for processing, and outputting a background image in which defects in the to-be-detected image are removed, and taking the background image as the target image; The determining of the defect region in the target image comprises: comparing pixel similarity of the to-be-detected image and the target image; regarding a pixel block formed by pixels with a pixel similarity lower than a similarity threshold in the target image as a defect region.

7. The method of claim 6, wherein, After the pixel block composed of the pixels in the target image with pixel similarity lower than the similarity threshold is taken as the defect region, the method further comprises: performing filtering processing on the target image by using a pre-stored morphological filtering algorithm to obtain an optimized image, wherein the optimized image does not include the interference defect region.

8. The method of claim 1, wherein, The to-be-detected image includes an image of a to-be-detected printed circuit board acquired by a detection machine, the detection machine is a printed circuit board detection device in a printed circuit board production line, and the defect region corresponds to a position of an industrial defect in the to-be-detected printed circuit board.

9. A defect detection apparatus characterized by comprising: The method comprises the following steps: a to-be-detected image acquisition module is configured to acquire a to-be-detected image; a to-be-detected image processing module is configured to input the to-be-detected image into a pre-stored attention generative adversarial network for processing to output a target image; the attention generative adversarial network comprises an attention mask generator and a content mask generator, the attention mask generator is configured to generate a foreground attention mask and a background attention mask, and the content mask generator is configured to generate a content mask; the attention generative adversarial network is trained by using an unpaired source domain image set and a target domain image set, the images in the source domain image set are images of detected objects with defects, the images in the target domain image set are images of detected objects without defects, the target image is an attention mask image output by the attention generative adversarial network or a background image with defects removed from the to-be-detected image; a defect region determination module is configured to determine a defect region in the target image.

10. An electronic device, comprising: The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 8.

Images