Pattern defect detection method and device and storage medium

Abstract

The invention discloses a pattern defect detection method and device and a storage medium, and the method comprises the steps: obtaining an overall pattern region and a plurality of local pattern regions corresponding to a target pattern in a to-be-detected image, and obtaining an overall template image and a plurality of local template images from the template information corresponding to the target pattern; aligning the overall pattern area with the overall template image, and respectively aligning each local pattern area with the corresponding local template image; determining a first defect position in the overall pattern area based on a first pixel value difference between the aligned overall pattern area and the overall template image, and determining a second defect position in each local pattern area based on a second pixel value difference between each aligned local pattern area and the corresponding local template image; and synthesizing the first defect position and each second defect position, and determining a third defect position of the target pattern in the to-be-detected image. Through the method, the accuracy of pattern defect detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision, and in particular to a pattern defect detection method, device and storage medium. Background Art

[0002] In the field of machine vision, for fixed patterns or character sequences, when there is deformation in the image, the abnormal area in the image can be detected using the pattern defect detection method. However, the existing defect detection algorithm has poor adaptability to complex situations such as distortion or irregular deformation in the image, resulting in low accuracy of defect detection. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a pattern defect detection method, device and medium, which can improve the accuracy of pattern defect detection.

[0004] In order to solve the above technical problems, a technical solution adopted in the present application is to provide a pattern defect detection method, which includes: obtaining an overall pattern area and several local pattern areas corresponding to a target pattern in an image to be detected, and obtaining an overall template image and several local template images from the template information corresponding to the target pattern, wherein each of the local pattern areas corresponds to a different local area of ​​the target pattern, the overall template image and the local template image are respectively the overall image and the local image of the target pattern in a defect-free state, and each of the local pattern areas corresponds to each of the local template images one by one; aligning the overall pattern area with the overall template image, and aligning each of the local pattern areas with the corresponding local template image respectively; determining a first defect position in the overall pattern area based on a first pixel value difference between the aligned overall pattern area and the overall template image, and determining a second defect position in each of the local pattern areas based on a second pixel value difference between each of the aligned local pattern areas and the corresponding local template image; and determining a third defect position of the target pattern in the image to be detected based on the first defect position and each of the second defect positions.

[0005] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an electronic device, the electronic device includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the above pattern defect detection method.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above pattern defect detection method.

[0007] The above scheme aligns the whole pattern area with the whole template image, and aligns each local pattern area with the corresponding local template image. Based on the first pixel value difference between the aligned whole pattern area and the whole template image, the first defect position in the whole pattern area is determined, and based on the second pixel value difference between each aligned local pattern area and the corresponding local template image, the second defect position in each local pattern area is determined. The third defect position of the target pattern in the image to be detected is determined by combining the first defect position and each second defect position. The present application can identify the first defect position in the whole pattern area based on the first pixel value difference between the aligned whole pattern area and the whole template image. For each local pattern area, the second pixel value difference between it and the corresponding local template image is also calculated to determine the second defect position in each local pattern area. The first defect position and the second defect position are comprehensively analyzed, and the third defect position of the target pattern in the image to be detected is obtained by integrating defect information of different dimensions. When processing complex situations such as distortion or irregular deformation in the image, the present application adopts a block detection method to achieve in-depth analysis of the details of distortion or irregular deformation. At the same time, the whole image detection method is used to analyze the overall condition of the image. This combination of whole image detection and block detection enables the present application to more comprehensively capture and accurately locate defects in the target pattern, thereby improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of an embodiment of a pattern defect detection method provided by the present application;

[0009] Figure 2 It is a schematic diagram of the training process of the defect detection model of the pattern defect detection method provided in this application;

[0010] Figure 3 It is a partial flow chart of a specific embodiment of the pattern defect detection method provided by the present application;

[0011] Figure 4 It is a flow chart of an embodiment of a pattern defect detection device of the present application;

[0012] Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0013] Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0015] It should be noted that the term "several" in this article means at least one, and the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of types. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] See also Figure 1 , Figure 1 is a flow chart of an embodiment of a pattern defect detection method provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0017] Step S11: obtaining an overall pattern region and several local pattern regions corresponding to a target pattern in an image to be detected, and obtaining an overall template image and several local template images from template information corresponding to the target pattern.

[0018] Among them, each local pattern area corresponds to a different local area of ​​the target pattern, the overall template image and the local template image are the overall image and the local image of the target pattern in a defect-free state, respectively, and each local pattern area corresponds to each local template image one by one.

[0019] The image to be detected can be obtained by shooting with professional equipment, uploading by users, or using public data sets. Before obtaining the overall pattern area and several local pattern areas corresponding to the target pattern in the image to be detected, the image to be detected can be preprocessed by image enhancement, noise reduction, filtering, morphological transformation, image smoothing, or image registration and alignment to improve the quality of the image to be detected.

[0020] In a specific embodiment, before obtaining the overall pattern area and several local pattern areas corresponding to the target pattern in the image to be detected, it also includes: preprocessing the image to be detected, wherein the preprocessing includes at least one of the following: noise reduction, filtering and morphological transformation.

[0021] The image to be detected contains a target pattern for pattern defect detection. The target pattern contained in the image to be detected can be obtained by using deep learning, template matching, etc. After the target pattern is obtained, the target pattern can be kept consistent with the template image in terms of size, color, position, format, etc. by using operations such as resizing, color correction, image registration, image enhancement, and format conversion.

[0022] In one embodiment, obtaining the overall pattern region includes: performing target detection on the image to be detected using a target detection model to obtain the overall pattern region; or performing template matching on the image to be detected using an overall template image corresponding to the target pattern to obtain the overall pattern region. In response to the size of the overall pattern region being inconsistent with the overall template image, the size of the overall pattern region is adjusted to be consistent with the overall template image.

[0023] The local pattern area can be obtained from the local pattern uploaded by the user or photographed by professional equipment, or can be obtained by dividing the overall pattern area or the target pattern into blocks. In one embodiment, the target pattern can be divided into blocks using overlapping sliding windows, uniform segmentation, deep learning, etc. to obtain the corresponding local pattern area. In a specific embodiment, obtaining several local pattern areas includes: dividing the overall pattern area into blocks in an overlapping sliding window manner to obtain several local pattern areas, wherein the window of the overlapping sliding window is the same size as the local template image, and the overlapping ratio of the overlapping sliding window is preset or input by the user.

[0024] Step S12: aligning the overall pattern area with the overall template image, and aligning each local pattern area with the corresponding local template image.

[0025] Alignment in this article refers to the process of matching feature points or feature areas of an image (whole pattern area or local pattern area) with another image (whole template image or local template image) through a certain transformation method. In one embodiment, the whole pattern area is aligned with the whole template image, or each local pattern area is aligned with the corresponding local template image, including: obtaining alignment fine-grained information from the template information. According to the alignment fine-grained information, the whole pattern area is aligned with the whole template image, or the local pattern area is aligned with the corresponding local template image.

[0026] Before alignment, the overall pattern area or the local pattern area may be preprocessed to improve the accuracy and efficiency of alignment. In one embodiment, before aligning the overall pattern area with the overall template image and aligning the local template images corresponding to the local pattern areas, feature point information corresponding to the overall pattern area and the local pattern areas may be obtained. For example, the feature point information may be a template matching result or a deep learning feature map.

[0027] In another embodiment, before aligning the overall pattern area with the overall template image, and aligning the local template images corresponding to each local pattern area, the overall pattern area or each local pattern area can be subjected to feature point detection and feature point filtering and other processing to reduce unnecessary feature points. In a specific embodiment, before aligning the overall pattern area with the overall template image, and aligning each local pattern area with the corresponding local template image, it also includes: finding the local pattern area with a number of feature points less than a preset number as the area to be merged, and merging the area to be merged with the adjacent local pattern area as the new local pattern area. The local template image corresponding to the area to be merged and the local template image corresponding to the adjacent local pattern area are merged as a new local template image. Among them, the preset number can be set by the user, or dynamically adjusted according to actual needs.

[0028] For example, the preset number of feature points is 4. The number of feature points corresponding to the local pattern area is obtained, wherein the number of feature points can be obtained from the feature point information. When the number of feature points in a local pattern area is less than 4, the local pattern area is merged with an adjacent local pattern area as a new local pattern area. When the number of feature points corresponding to the new local pattern area is greater than 4, the new local pattern area is used as a local pattern area for subsequent alignment with the local template image. When the number of feature points corresponding to the new local pattern area is not greater than 4, the operation of merging the local pattern area with the adjacent local pattern area is repeated until the number of feature points of the new local pattern area is greater than 4.

[0029] In another embodiment, before alignment, the distortion, blur and other unstable factors in the overall pattern area or the local pattern area can be reduced by correction. Among them, the overall pattern area can be corrected, and each local pattern area can be corrected separately by using a feature-based correction method, template matching, iterative nearest point algorithm or deep learning. For example, using a feature-based correction method, the overall mapping relationship between the overall pattern area and the overall template image is obtained, and the local mapping relationship between each local pattern area and the corresponding local template image is obtained, the overall pattern area and the overall template image are corrected by using the overall mapping relationship, and each local pattern area and the corresponding local template image are corrected by using the local mapping relationship.

[0030] In a specific embodiment, obtaining a target mapping relationship between a target pattern area and a target template image includes: randomly selecting multiple groups of feature point pairs from the target image area and the target template image, and determining the initial mapping relationship between the target pattern area and the target template image based on the multiple groups of feature point pairs. Repeat this step to obtain multiple initial mapping relationships. Obtain the central tendency characterization value of the multiple initial mapping relationships as the target mapping relationship. Wherein, the target pattern area is the overall pattern area, and the target mapping relationship is the overall mapping relationship, or, the target pattern area is the local pattern area, and the target mapping relationship is the local mapping relationship. The central tendency characterization value is used to characterize the central position or general level of multiple initial mapping relationships, and can be a mean, median, mode, etc., which is not limited here.

[0031] For example, according to the corresponding number of feature points between the target pattern area and the target template image and the RANSAC (random sampling consensus) method, multiple sets of feature point pairs are randomly selected from the target image area and the target template image, and the initial mapping relationship between the target pattern area and the target template image is determined by the multiple sets of feature point pairs. The number of feature points contained in each set of feature point pairs is consistent. Repeat this step to obtain multiple initial mapping relationships. The average mapping relationship of the multiple initial mapping relationships is obtained as the target mapping relationship.

[0032] Among them, before randomly selecting multiple groups of feature point pairs from the target image area and the target template image, or finding a local pattern area with a number of feature points less than a preset number as the area to be merged, the feature points can be filtered. In a specific embodiment, before randomly selecting multiple groups of feature point pairs from the target image area and the target template image, it also includes: filtering the feature points in the target image area and the target template area based on the number of pixels of the target template image.

[0033] For example, based on the number of pixels of the target template image, the corresponding number of feature points in the target template image is obtained. Among them, for those areas where the feature points are densely distributed (that is, the blocks with a large number of feature points and richer information in the corresponding areas of the target template image), you can choose to retain relatively more feature points. These feature points can be points at the middle or edge positions of the area. For those areas where the feature points are sparsely distributed (that is, the areas with a small number of feature points and poorer information in the corresponding areas of the target template image), you can choose to reduce the number of retained feature points. Randomly select multiple groups of feature point pairs from the feature points retained in the target image area and the target template image. Among them, the specific feature point filtering operation can be adjusted according to actual needs, and there is no restriction here.

[0034] Step S13: Determine the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determine the second defect position in each local pattern area based on the second pixel value difference between each local pattern area and the corresponding local template image after alignment.

[0035] The first pixel value difference and the second pixel value difference in this embodiment can be obtained by using a difference image, threshold segmentation, differential method, template matching, decision tree or deep learning. In one embodiment, based on the first pixel value difference between the aligned overall pattern area and the overall template image, the first defect position in the overall pattern area is determined, or based on the second pixel value difference between each aligned local pattern area and the corresponding local template image, the second defect position in each local pattern area is determined, including: obtaining a first difference image between the target template image and the target pattern area, wherein each pixel value in the first difference image represents the target pixel value difference between each pixel point in the target pattern area and the corresponding pixel point in the target template image. From the first difference image, a number of first target pixel points whose pixel values ​​are greater than a preset difference threshold are found, wherein the number of first target pixel points are used to define the first target defect position. Among them, the target template image is the overall template image, the target pattern area is the overall pattern area, the target pixel value difference is the first pixel value difference, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, the target pixel value difference is the second pixel value difference, and the first target defect position is the second defect position.

[0036] In a specific embodiment, a plurality of first target pixel points are used to define the first target defect position. The plurality of first target pixel points may be regarded as indicator points of the first target defect position, and these first target pixel points are connected or expanded with the first target pixel point as the center to form a small area, which is used as the first target defect position. Alternatively, a plurality of first target pixel points may be screened, and those first target pixel points that are single, isolated, or do not meet the defect characteristics may be deleted or ignored, and the area where the screened first target pixel points are located may be used as the first target defect position. The specific method of using a plurality of first target pixel points to define the first target defect position may be adjusted according to the actual situation and is not limited here.

[0037] The corresponding pixel point can be determined as the first target pixel point by finding the relationship between the pixel value in the first difference image and the preset difference threshold, or the pixel point with abnormal change rate can be determined as the first target pixel point by calculating the change rate of the pixel values ​​of adjacent pixels or a specific area in the first difference image. In one embodiment, a number of first target pixel points whose pixel values ​​are greater than the preset difference threshold are found from the first difference image, including: obtaining a threshold image corresponding to the target template image from the template information. The pixel value of the second target pixel point in the first difference image is set to an invalid value to obtain a second difference image, wherein the pixel value of the second target pixel point is not greater than the pixel value of the corresponding pixel point in the threshold image. A number of first target pixel points whose pixel values ​​are greater than the preset difference threshold are found from the second difference image.

[0038] In another specific embodiment, obtaining a first difference image between the target template image and the target pattern region includes: obtaining a standard deviation image corresponding to the target template image from the template information. Obtaining a difference image between the target pattern region and the standard deviation image as the first difference image.

[0039] Before obtaining the first difference image, the target pattern area may be normalized, standardized, or linearly stretched to obtain an accurate difference map between the target image area and the standard deviation image. In one embodiment, before obtaining the first difference image between the target template image and the target pattern area, it also includes: obtaining a normalization method from the template information. The target pattern area is normalized using the normalization method, wherein the target image area after the normalization is used to obtain the first difference image.

[0040] Taking the determination of the first target pixel as an example, a normalization method is obtained from the template information in the overall template image. The overall pattern area is normalized using the normalization method to obtain an overall normalized image. A difference map between the standard deviation image in the template information and the overall normalized image is obtained as the first difference image. The threshold image in the template information is obtained, and the pixel points in the first difference image whose pixel values ​​exceed the pixel values ​​in the threshold image are used as the second target pixel points. The pixel values ​​corresponding to the second target pixel points in the first difference image are set to invalid values ​​to obtain a second difference image. The area with larger differences in the second difference image is obtained, and the pixel points corresponding to the area with larger differences are used as the first target pixel points.

[0041] For example, a threshold image is derived from a target template image using a linear transformation, wherein a linear transformation parameter (such as a scaling factor or a translation factor) can determine the pixel value range of the threshold image and adjust the contrast and brightness in the image. By setting the pixel points in the first difference image whose pixel values ​​are not greater than the pixel values ​​of the corresponding pixel points in the threshold image to 0 or a specific label value, the second difference image contains pixel points that are significantly different from the target template image. The pixel points in the second difference image whose pixel values ​​are greater than a preset difference threshold are determined as the first target pixel points.

[0042] Before determining the first defect position or the second defect position based on the first pixel value difference or the second pixel value difference, a deformation area with a large difference can be found, and the deformation area can be directly output as the first defect position or the second defect position. In one embodiment, before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each local pattern area based on the second pixel value difference between each aligned local pattern area and the corresponding local template image, it also includes: obtaining a target mapping relationship between the target pattern area and the target template image. In response to determining that the deformation degree of the target pattern area is greater than the preset degree threshold based on the target mapping relationship, a deformation area in the target image area whose deformation degree meets the deformation requirement is determined, and the deformation area is used as the second target defect position, and the step of directly performing the combination of the first defect position and each second defect position to determine the third defect position of the target pattern in the image to be detected is performed. Among them, the target template image is the overall template image, the target pattern area is the overall pattern area, and the second target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the second target defect position is the second defect position.

[0043] In another embodiment, before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each local pattern area based on the second pixel value difference between each local pattern area and the corresponding local template image after alignment, it also includes: obtaining a target mapping relationship between the target pattern area and the target template image. In response to determining that the target pattern area is deformed based on the target mapping relationship and the degree of deformation is not greater than a preset degree threshold, the target pattern area is corrected using the target mapping relationship, wherein the corrected target pattern area is used for alignment and determining the first target defect position in the target pattern area; before determining the third defect position of the target pattern in the image to be detected by combining the first defect position and each second defect position, it also includes: mapping the first target defect position to the target image area before correction using the target mapping relationship. Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the first target defect position is the second defect position.

[0044] The method of obtaining the target mapping relationship may be specifically described in reference to the relevant description in step S12.

[0045] Taking the determination of the second defect position as an example, the local mapping relationship between each local pattern area and the local template image is obtained. Based on the relevant parameters in each local mapping relationship, it is judged whether each local pattern area is deformed and the degree of deformation is not greater than the degree threshold, for example, the judgment is made using relevant parameters such as deformation gradient and deformation volume ratio. In the case of whether a local pattern area is deformed and the degree of deformation is not greater than the degree threshold, the local pattern area is corrected using the corresponding local mapping relationship to obtain the corrected local pattern area. The corrected local pattern area is aligned with the corresponding local template image. Based on the second pixel value difference between the aligned local pattern area and the local template image, the second defect position in the local pattern area is determined. Using the corresponding local mapping relationship and the second difference image, the second defect position is mapped to the local pattern area before correction. In the case of whether a local pattern area is deformed and the degree of deformation is greater than the degree threshold, the local pattern area is directly output as the second defect position.

[0046] Step S14: Determine a third defect position of the target pattern in the image to be detected by combining the first defect position and each second defect position.

[0047] The first defect position and the second defect position may be combined by weighting, edge detection, morphological operation, image registration, etc. to determine a third defect position of the target pattern in the image to be detected. In one embodiment, combining the first defect position and each second defect position to determine the third defect position of the target pattern in the image to be detected includes: weighting the first defect position and each second defect position to obtain the third defect position.

[0048] In a specific embodiment, the second defect positions can be integrated to obtain the local defect position of the target pattern, wherein the local defect position can be used to characterize the overall result of the local detection, and the local defect position of the target pattern is weighted with the first defect position to obtain the third defect position.

[0049] In one embodiment, the first defect location, the second defect location, and the third defect location are all characterized by using a mask image.

[0050] In another implementation manner, this embodiment is implemented by a defect detection model, and the template information is learned by the defect detection model during a training process using defect-free pattern samples corresponding to the target pattern.

[0051] Combination Figure 2 To illustrate with an example, Figure 2It is a schematic diagram of the training process of the defect detection model of the pattern defect detection method provided in the present application. Collect defect-free pattern samples, which can contain the allowable defect error range as much as possible, wherein the defect-free pattern samples come from one or more defect-free images. Usually, multiple images are selected, which can suppress the differences caused by uneven illumination to a certain extent. Preprocess the defect-free pattern samples, including noise reduction, filtering, and morphological transformation, to ensure the consistency of image quality. Obtain the standard deviation image, threshold image, and normalized image of the defect-free pattern sample, set the linear transformation parameters in the threshold image and the normalization method of the image. Obtain feature information, which is used for alignment, comparison, screening of fine-grained matching results, mask image generation, etc. between templates. The mask image can be a binary image, in which the defect area is marked as 1 (or white) and the non-defect area is marked as 0 (or black). During the training process, the parameters and structure of the defect detection model can be continuously optimized to improve the accuracy and robustness of defect detection. Through a large number of defect-free pattern sample inputs and repeated training iterations, the defect detection model can learn to capture the feature information in the image to be detected and achieve accurate detection of pattern defects. Based on the above-mentioned template training method, the corresponding overall template and local template are trained, and the corresponding overall template image or local template image can be obtained from the overall template or local template. Among them, the overall template image and the local template image both include two parts: template image and feature information. The template image includes a standard deviation image, a threshold image, and a normalized image calculated using a defect-free image. The feature information is used for alignment, comparison, screening of fine-grained matching results, and mask image generation between templates.

[0052] This embodiment aligns the overall pattern area with the overall template image, and aligns each local pattern area with the corresponding local template image. Based on the first pixel value difference between the aligned overall pattern area and the overall template image, the first defect position in the overall pattern area is determined, and based on the second pixel value difference between each aligned local pattern area and the corresponding local template image, the second defect position in each local pattern area is determined. The third defect position of the target pattern in the image to be detected is determined by combining the first defect position and each second defect position. The present application can identify the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image. For each local pattern area, the second pixel value difference between it and the corresponding local template image is also calculated to determine the second defect position in each local pattern area. The first defect position and the second defect position are comprehensively analyzed, and the third defect position of the target pattern in the image to be detected is obtained by integrating defect information of different dimensions. When processing complex situations such as distortion or irregular deformation in the image, the present application adopts a block detection method to achieve in-depth analysis of the details of distortion or irregular deformation. At the same time, the whole image detection method is used to analyze the overall condition of the image. This combination of whole image detection and block detection enables the present application to more comprehensively capture and accurately locate defects in the target pattern, thereby improving the accuracy of defect detection.

[0053] See also Figure 3 , Figure 3 FIG. 1 is a partial flow chart of a specific embodiment of a pattern defect detection method provided by the present application. Figure 3 As shown, the pattern defect detection method provided in this embodiment may include the following steps:

[0054] Step S31: Obtain the overall pattern area of ​​the target pattern in the image to be detected. Step S32: According to the feature information of the target pattern, the target pattern is divided into blocks to obtain several local pattern areas. Step S33: Align the overall pattern area with the overall template image, and align each local pattern area with the corresponding local template image. Step S34: Extract the overall feature information in the overall pattern area, and extract the local feature information in each local pattern area. Step S35: Use the overall feature information to match and compare with the template image in the overall template image, and use each local feature information to match and compare with the template image in the corresponding local template image. Step S36: Determine whether the matching is successful. If the matching is successful, execute step S37: Combine the first defect position and the second defect position to output the third defect position. If the matching fails, execute step S38: Determine that the target pattern defect is large and invalid.

[0055] See also Figure 4 , Figure 4 It is a flow chart of an embodiment of the pattern defect detection device of the present application. The pattern defect detection device 400 includes a pattern acquisition module 410, an alignment module 420, a defect position acquisition module 430, and a defect position synthesis module 440. The pattern acquisition module 410 is used to acquire the overall pattern area and several local pattern areas corresponding to the target pattern in the image to be detected, and to acquire the overall template image and several local template images from the template information corresponding to the target pattern, wherein each local pattern area corresponds to a different local area of ​​the target pattern, the overall template image and the local template image are respectively the overall image and the local image of the target pattern in a defect-free state, and each local pattern area corresponds to each local template image one by one. The alignment module 420 is used to align the overall pattern area with the overall template image, and to align each local pattern area with the corresponding local template image respectively. The defect position acquisition module 430 is used to determine the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and to determine the second defect position in each local pattern area based on the second pixel value difference between each aligned local pattern area and the corresponding local template image. The defect position synthesis module 440 is used to synthesize the first defect position and each second defect position to determine a third defect position of the target pattern in the image to be detected.

[0056] In some embodiments, the defect position acquisition module 430 determines the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, or determines the second defect position in each local pattern area based on the second pixel value difference between each local pattern area and the corresponding local template image after alignment, including: acquiring a first difference image between the target template image and the target pattern area, wherein each pixel value in the first difference image represents the target pixel value difference between each pixel point in the target pattern area and the corresponding pixel point in the target template image. From the first difference image, find out a number of first target pixel points whose pixel values ​​are greater than a preset difference threshold, wherein the number of first target pixel points are used to define the first target defect position. Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, the target pixel value difference is the first pixel value difference, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, the target pixel value difference is the second pixel value difference, and the first target defect position is the second defect position.

[0057] In some embodiments, the defect location acquisition module 430 performs the following steps when searching for a number of first target pixel points whose pixel values ​​are greater than a preset difference threshold from the first difference image: obtaining a threshold image corresponding to the target template image from the template information. Setting the pixel value of the second target pixel point in the first difference image to an invalid value to obtain a second difference image, wherein the pixel value of the second target pixel point is not greater than the pixel value of the corresponding pixel point in the threshold image. Searching for a number of first target pixel points whose pixel values ​​are greater than the preset difference threshold from the second difference image.

[0058] In some embodiments, the defect location acquisition module 430 acquires the first difference image between the target template image and the target pattern area by: acquiring a standard deviation image corresponding to the target template image from the template information, and acquiring a difference image between the target image area and the standard deviation image as the first difference image.

[0059] In some embodiments, before the defect location acquisition module 430 executes the acquisition of the first difference image between the target template image and the target pattern area, it also includes: acquiring a normalization method from the template information. The target pattern area is normalized using the normalization method, wherein the target image area after the normalization is used to acquire the first difference image.

[0060] In some embodiments, the pattern defect detection device 400 further includes a deformation degree detection module. Before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each local pattern area based on the second pixel value difference between each local pattern area and the corresponding local template image after alignment, the deformation degree detection module includes: obtaining a target mapping relationship between the target pattern area and the target template image. In response to determining that the degree of deformation of the target pattern area is greater than a preset degree threshold based on the target mapping relationship, determining a deformed area in the target image area whose degree of deformation meets the deformation requirement, taking the deformed area as the second target defect position, and directly performing the step of combining the first defect position and each second defect position to determine the third defect position of the target pattern in the image to be detected. Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, and the second target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the second target defect position is the second defect position.

[0061] In some embodiments, the pattern defect detection device 400 further includes a deformation degree detection module. Before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each local pattern area based on the second pixel value difference between each aligned local pattern area and the corresponding local template image, the deformation degree detection module includes: obtaining a target mapping relationship between the target pattern area and the target template image. In response to determining that the target pattern area is deformed and the degree of deformation is not greater than a preset degree threshold based on the target mapping relationship, the target pattern area is corrected using the target mapping relationship, wherein the corrected target pattern area is used for alignment and determining the first target defect position in the target pattern area. Before determining the third defect position of the target pattern in the image to be detected by combining the first defect position and each second defect position, it also includes: mapping the first target defect position to the target image area before correction using the target mapping relationship. Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the first target defect position is the second defect position.

[0062] In some embodiments, the deformation degree detection module performs the step of obtaining a target mapping relationship between a target pattern area and a target template image, including: randomly selecting multiple sets of feature point pairs from the target image area and the target template image, and determining an initial mapping relationship between the target pattern area and the target template image from the multiple sets of feature point pairs. Repeating this step to obtain multiple initial mapping relationships. Obtaining a central tendency characterization value of the multiple initial mapping relationships as the target mapping relationship.

[0063] In some embodiments, the pattern defect detection device 400 further includes a block division module. Before aligning the overall pattern area with the overall template image and aligning each local pattern area with the corresponding local template image, the block division module includes: finding a local pattern area with a number of feature points less than a preset number as a to-be-merged area, and merging the to-be-merged area with an adjacent local pattern area as a new local pattern area. Merging the local template image corresponding to the to-be-merged area and the local template image corresponding to the adjacent local pattern area as a new local template image.

[0064] In some embodiments, the deformation degree detection module or the blocking module, before randomly selecting multiple groups of feature point pairs from the target image area and the target template image, or finding a local pattern area with a number of feature points less than a preset number as the area to be merged, also includes: filtering the feature points in the target image area and the target template area based on the number of pixels of the target template image.

[0065] In some embodiments, the alignment module 420 performs the alignment of the overall pattern area with the overall template image, or aligns each local pattern area with the corresponding local template image, including: obtaining alignment fine-grained information from the template information. According to the alignment fine-grained information, the overall pattern area is aligned with the overall template image, or the local pattern area is aligned with the corresponding local template image.

[0066] In some embodiments, the defect location synthesis module 440 performs synthesis of the first defect location and each second defect location to determine the third defect location of the target pattern in the image to be detected, including: weighting the first defect location and each second defect location to obtain the third defect location.

[0067] In some embodiments, the pattern acquisition module 410 performs the acquisition of the overall pattern area, including: using the target detection model to perform target detection on the image to be detected to obtain the overall pattern area. Alternatively, using the overall template image corresponding to the target pattern to perform template matching on the image to be detected to obtain the overall pattern area. In response to the size of the overall pattern area being inconsistent with the overall template image, the size of the overall pattern area is adjusted to be consistent with the overall template image.

[0068] In some embodiments, the pattern acquisition module 410 acquires several local pattern areas by dividing the overall pattern area into blocks in an overlapping sliding window manner to obtain several local pattern areas, wherein the window of the overlapping sliding window is the same size as the local template image, and the overlapping ratio of the overlapping sliding window is preset or input by the user.

[0069] In some embodiments, before executing acquisition of the overall pattern area and several local pattern areas corresponding to the target pattern in the image to be detected, the pattern acquisition module 410 further includes: preprocessing the image to be detected, wherein the preprocessing includes at least one of the following: noise reduction, filtering and morphological transformation.

[0070] In some embodiments, the pattern defect detection method is implemented by a defect detection model, and the template information is learned by the defect detection model during the training process using defect-free pattern samples corresponding to the target pattern.

[0071] In some embodiments, the first defect location, the second defect location, and the third defect location are all characterized by using a mask image.

[0072] See also Figure 5 , Figure 5: is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above pattern defect detection method embodiments. In a specific implementation scenario, the electronic device 50 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0073] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned pattern defect detection method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0074] See also Figure 6 , Figure 6 The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any of the above-mentioned pattern defect detection method embodiments.

[0075] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0076] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0077] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A pattern defect detection method, characterized in that: The method comprises: Acquire an overall pattern region and several local pattern regions corresponding to a target pattern in an image to be detected, and acquire an overall template image and several local template images from template information corresponding to the target pattern, wherein each of the local pattern regions corresponds to a different local region of the target pattern, the overall template image and the local template image are respectively an overall image and a local image of the target pattern in a defect-free state, and each of the local pattern regions corresponds to each of the local template images one by one; Aligning the overall pattern area with the overall template image, and respectively aligning each of the local pattern areas with the corresponding local template image; Determine a first defect position in the overall pattern area based on a first pixel value difference between the aligned overall pattern area and the overall template image, and determine a second defect position in each of the local pattern areas based on a second pixel value difference between each of the aligned local pattern areas and the corresponding local template image; The first defect position and each of the second defect positions are combined to determine a third defect position of the target pattern in the image to be detected.

2. The method according to claim 1, characterized in that The determining of the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, or the determining of the second defect position in each of the local pattern areas based on the second pixel value difference between each of the aligned local pattern areas and the corresponding local template image, comprises: Acquire a first difference image between a target template image and a target pattern region, wherein each pixel value in the first difference image represents a target pixel value difference between each pixel point in the target pattern region and a corresponding pixel point in the target template image; Finding a plurality of first target pixel points whose pixel values ​​are greater than a preset difference threshold from the first difference image, wherein the plurality of first target pixel points are used to define a first target defect position; Among them, the target template image is the overall template image, the target pattern area is the overall pattern area, the target pixel value difference is the first pixel value difference, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, the target pixel value difference is the second pixel value difference, and the first target defect position is the second defect position.

3. The method according to claim 2, characterized in that The step of searching for a plurality of first target pixel points whose pixel values ​​are greater than a preset difference threshold from the first difference image includes: Acquire a threshold image corresponding to the target template image from the template information; Setting the pixel value of the second target pixel point in the first difference image to an invalid value to obtain a second difference image, wherein the pixel value of the second target pixel point is not greater than the pixel value of the corresponding pixel point in the threshold image; The first target pixel points whose pixel values ​​are greater than the preset difference threshold are found from the second difference image.

4. The method according to claim 2, characterized in that: The step of acquiring a first difference image between a target template image and a target pattern area includes: Acquire a standard deviation image corresponding to the target template image from the template information; Acquire a difference image between the target pattern area and the standard deviation image as the first difference image; And / or, before acquiring the first difference image between the target template image and the target pattern area, the method further includes: Acquire a normalization method from the template information; The target pattern area is normalized by using the normalization method, wherein the target image area after the normalization is used to obtain the first difference image.

5. The method according to claim 1, characterized in that Before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each of the local pattern areas based on the second pixel value difference between each of the aligned local pattern areas and the corresponding local template image, the method further includes: Acquire a target mapping relationship between a target pattern area and a target template image; In response to determining, based on the target mapping relationship, that the degree of deformation of the target pattern area is greater than a preset degree threshold, determining a deformed area in the target image area whose degree of deformation meets the deformation requirement, taking the deformed area as a second target defect position, and directly performing the step of combining the first defect position and each of the second defect positions to determine a third defect position of the target pattern in the image to be detected; Among them, the target template image is the overall template image, the target pattern area is the overall pattern area, and the second target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the second target defect position is the second defect position.

6. The method according to claim 1, characterized in that Before determining the first defect position in the overall pattern area based on the first pixel value difference between the aligned overall pattern area and the overall template image, and determining the second defect position in each of the local pattern areas based on the second pixel value difference between each of the aligned local pattern areas and the corresponding local template image, the method further includes: Acquire a target mapping relationship between a target pattern area and a target template image; In response to determining that the target pattern area is deformed and the degree of deformation is not greater than a preset degree threshold based on the target mapping relationship, the target pattern area is corrected using the target mapping relationship, wherein the corrected target pattern area is used for the alignment and the determination of the first target defect position in the target pattern area; Before the step of combining the first defect position and each of the second defect positions to determine the third defect position of the target pattern in the image to be detected, the method further includes: Mapping the first target defect position to the target image region before correction using the target mapping relationship; Among them, the target template image is the overall template image, the target pattern area is the overall pattern area, and the first target defect position is the first defect position, or the target template image is the local template image, the target pattern area is the local pattern area, and the first target defect position is the second defect position.

7. The method according to claim 5 or 6, characterized in that: The obtaining of a target mapping relationship between a target pattern area and a target template image includes: Randomly select multiple groups of feature point pairs from the target image area and the target template image, and determine the initial mapping relationship between the target pattern area and the target template image using the multiple groups of feature point pairs; repeat this step to obtain multiple initial mapping relationships; The central tendency characterization values ​​of the multiple initial mapping relationships are obtained as the target mapping relationship.

8. The method according to claim 7, characterized in that Before randomly selecting a plurality of feature point pairs from the target image area and the target template image, the method further includes: Based on the number of pixels of the target template image, feature points in the target image area and the target template area are filtered.

9. The method according to claim 1, characterized in that: Before aligning the overall pattern area with the overall template image and respectively aligning each of the local pattern areas with the corresponding local template image, the method further includes: Find the local pattern area with a number of feature points less than a preset number as the area to be merged, and merge the area to be merged with the adjacent local pattern area as a new local pattern area; The local template image corresponding to the area to be merged and the local template image corresponding to the adjacent local pattern area are merged to form a new local template image.

10. The method according to claim 1, characterized in that The aligning the overall pattern area with the overall template image, or the aligning each of the local pattern areas with the corresponding local template image, comprises: Acquire alignment fine-grained information from the template information; According to the alignment fine-grained information, aligning the overall pattern area with the overall template image, or aligning the local pattern area with the corresponding local template image; And / or, the step of combining the first defect position and each of the second defect positions to determine a third defect position of the target pattern in the image to be detected includes: The first defect position and each of the second defect positions are weighted to obtain the third defect position.

11. The method according to claim 1, characterized in that: Acquiring the overall pattern area includes: Using a target detection model to perform target detection on the image to be detected to obtain the overall pattern area; or using an overall template image corresponding to the target pattern to perform template matching on the image to be detected to obtain the overall pattern area; In response to the size of the overall pattern area not being consistent with the size of the overall template image, adjusting the size of the overall pattern area to be consistent with the overall template image; And / or, obtaining the plurality of local pattern areas, comprising: Dividing the overall pattern area into blocks in an overlapping sliding window manner to obtain the plurality of local pattern areas, wherein the window of the overlapping sliding window is the same size as the local template image, and the overlapping ratio of the overlapping sliding window is preset or input by a user; And / or, before acquiring the overall pattern area and the plurality of local pattern areas corresponding to the target pattern in the image to be detected, the method further includes: The image to be detected is preprocessed, wherein the preprocessing includes at least one of the following: noise reduction, filtering and morphological transformation.

12. The method according to claim 1, characterized in that The method is implemented by a defect detection model, and the template information is learned by the defect detection model during the training process using defect-free pattern samples corresponding to the target pattern; And / or, the first defect position, the second defect position and the third defect position are all characterized by using a mask image.

13. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the pattern defect detection method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the pattern defect detection method according to any one of claims 1 to 12.

Images