A pattern defect detection method, device and storage medium

Abstract

The application discloses a pattern defect detection method and device and a storage medium. The method comprises the following steps: obtaining an overall pattern region and a plurality of local pattern regions corresponding to a target pattern in a to-be-detected image, and obtaining an overall template image and a plurality of local template images from template information corresponding to the target pattern; aligning the overall pattern region with the overall template image, and respectively aligning each local pattern region with a corresponding local template image; determining a first defect position in the overall pattern region based on a first pixel value difference between the aligned overall pattern region and the overall template image, and respectively determining a second defect position in each local pattern region based on a second pixel value difference between the aligned each local pattern region and the corresponding local template image; and comprehensively determining a third defect position of the target pattern in the to-be-detected image based on the first defect position and the second defect positions. Through the above method, the accuracy of pattern defect detection can be improved.

Description

Technical Field

[0001] This application relates to the field of machine vision, and more particularly to a method, device and storage medium for detecting pattern defects. Background Technology

[0002] In the field of machine vision, for fixed patterns or character sequences, abnormal regions in the image can be detected using pattern defect detection methods when the image is deformed. However, existing defect detection algorithms have poor adaptability to complex situations such as distortions or irregular deformations in the image, resulting in low accuracy in defect detection. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a method, device, and medium for detecting pattern defects, which can improve the accuracy of pattern defect detection.

[0004] To address the aforementioned technical problems, this application provides a pattern defect detection method. The method includes: acquiring an overall pattern region and several local pattern regions corresponding to a target pattern in an image to be detected; acquiring an overall template image and several local template images from template information corresponding to the target pattern; wherein each local pattern region corresponds to a different local region of the target pattern; the overall template image and the local template images are respectively the overall image and a local image of the target pattern in a defect-free state; each local pattern region corresponds one-to-one with each local template image; aligning the overall pattern region with the overall template image; and aligning each local pattern region with its corresponding local template image; determining a first defect position in the overall pattern region based on a first pixel value difference between the aligned overall pattern region and the overall template image; and determining a second defect position in each local pattern region based on a second pixel value difference between the aligned local pattern regions and their corresponding local template images; and combining the first defect position and each of the second defect positions to determine a third defect position of the target pattern in the image to be detected.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by this application is to provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor executes the program instructions to implement the above-mentioned pattern defect detection method.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed to implement the above-mentioned pattern defect detection method.

[0007] The above scheme aligns the overall pattern region with the overall template image, and aligns each local pattern region with its corresponding local template image. Based on the first pixel value difference between the aligned overall pattern region and the overall template image, the first defect location in the overall pattern region is determined. Similarly, based on the second pixel value difference between each aligned local pattern region and its corresponding local template image, the second defect location in each local pattern region is determined. Combining the first and second defect locations, the third defect location of the target pattern in the image to be detected is determined. This application can identify the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image. For each local pattern region, the second pixel value difference between it and its corresponding local template image is also calculated to determine the second defect location in each local pattern region. By comprehensively analyzing the first and second defect locations and integrating defect information from different dimensions, the third defect location of the target pattern in the image to be detected is obtained. When dealing with complex situations such as distortion or irregular deformation in the image, this application adopts a block-based detection method to achieve in-depth analysis of the details of distortion or irregular deformation. Simultaneously, a whole-image detection method is used to perform an overall analysis of the image's condition. This combination of whole-image detection and block detection enables this application to more comprehensively capture and accurately locate defects in the target pattern, thereby improving the accuracy of defect detection. Attached Figure Description

[0008] Figure 1 This is a schematic flowchart of an embodiment of the pattern defect detection method provided in this application;

[0009] Figure 2 This is a schematic diagram of the training process of the defect detection model of the pattern defect detection method provided in this application;

[0010] Figure 3 This is a partial flowchart of a specific embodiment of the pattern defect detection method provided in this application;

[0011] Figure 4 This is a schematic flowchart of an embodiment of the pattern defect detection device of this application;

[0012] Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0013] Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0014] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0015] It should be noted that in this article, the term "several" means at least one, and the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the pattern defect detection method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0017] Step S11: Obtain the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, and obtain the overall template image and several local template images from the template information corresponding to the target pattern.

[0018] Each local pattern region corresponds to a different local region of the target pattern. The overall template image and the local template image are the overall image and local image of the target pattern in a defect-free state, respectively. Each local pattern region corresponds one-to-one with each local template image.

[0019] Images to be detected can be obtained through methods such as professional equipment photography, user uploads, and public datasets. Before obtaining the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, the image to be detected can be preprocessed using methods such as image enhancement, noise reduction, filtering, morphological transformation, image smoothing, or image registration and alignment to improve the quality of the image to be detected.

[0020] In one specific embodiment, before acquiring the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, the method further includes: preprocessing the image to be detected, wherein the preprocessing includes at least one of the following: noise reduction, filtering, and morphological transformation.

[0021] The image to be inspected contains a target pattern used for pattern defect detection. This target pattern can be obtained using methods such as deep learning and template matching. After obtaining the target pattern, operations such as resizing, color correction, image registration, image enhancement, and format conversion can be used to ensure that the target pattern and the template image are consistent in size, color, position, and format.

[0022] In one embodiment, obtaining the overall pattern region includes: performing target detection on the image to be detected using a target detection model to obtain the overall pattern region; or, performing template matching on the image to be detected using an overall template image corresponding to the target pattern to obtain the overall pattern region. In response to a mismatch between the size of the overall pattern region and the overall template image, the size of the overall pattern region is adjusted to match that of the overall template image.

[0023] Local pattern regions can be obtained from user-uploaded or professionally captured images, or by dividing the overall pattern region or target pattern into blocks. In one embodiment, overlapping sliding windows, uniform segmentation, deep learning, or other methods can be used to segment the target pattern to obtain corresponding local pattern regions. In a specific embodiment, obtaining several local pattern regions includes: dividing the overall pattern region into blocks using overlapping sliding windows to obtain several local pattern regions, wherein the window size of the overlapping sliding window is the same as the size of the local template image, and the overlap ratio of the overlapping sliding windows is preset or input by the user.

[0024] Step S12: Align the overall pattern area with the overall template image, and align each local pattern area with its corresponding local template image.

[0025] In this paper, alignment refers to the process of matching feature points or feature regions between one image (an overall pattern region or a local pattern region) and another image (an overall template image or a local template image) through a certain transformation method. In one embodiment, aligning the overall pattern region with the overall template image, or aligning each local pattern region with its corresponding local template image, includes: obtaining fine-grained alignment information from template information; and aligning the overall pattern region with the overall template image, or aligning the local pattern region with its corresponding local template image, according to the fine-grained alignment information.

[0026] Before alignment, preprocessing operations can be performed on the overall pattern region or local pattern regions to improve alignment accuracy and efficiency. In one embodiment, before aligning the overall pattern region with the overall template image, and before aligning the local template images corresponding to each local pattern region, feature point information corresponding to the overall pattern region and several local pattern regions can be obtained. For example, the feature point information can be template matching results or deep learning feature maps.

[0027] In another embodiment, before aligning the overall pattern region with the overall template image, and before aligning the local template images corresponding to each local pattern region, feature point detection and feature point filtering can be performed on the overall pattern region or each local pattern region to reduce unnecessary feature points. In a specific embodiment, before aligning the overall pattern region with the overall template image, and before aligning each local pattern region with its corresponding local template image, the method further includes: identifying local pattern regions with fewer than a preset number of feature points as regions to be merged, and merging these regions with adjacent local pattern regions to form new local pattern regions. The local template image corresponding to the region to be merged and the local template images corresponding to adjacent local pattern regions are then merged to form a new local template image. The preset number can be set by the user or dynamically adjusted according to actual needs.

[0028] For example, the preset number of feature points is 4. The number of feature points corresponding to a local pattern region is obtained, where the number of feature points can be obtained from the feature point information. If the number of feature points in a local pattern region is less than 4, this local pattern region is merged with adjacent local pattern regions to form a new local pattern region. If the number of feature points in this new local pattern region is greater than 4, this new local pattern region is used as the local pattern region for subsequent alignment with the local template image. If the number of feature points in this new local pattern region is not greater than 4, the operation of merging this local pattern region with adjacent local pattern regions is repeated until the number of feature points in the new local pattern region is greater than 4.

[0029] In another implementation, before alignment, distortion, blurring, and other unstable factors in the overall pattern region or local pattern regions can be reduced through correction. This can be achieved using feature-based correction methods, template matching, iterative nearest-point algorithms, or deep learning to correct both the overall pattern region and each local pattern region individually. For example, a feature-based correction method can be used to obtain the overall mapping relationship between the overall pattern region and the overall template image, and the local mapping relationship between each local pattern region and its corresponding local template image. The overall mapping relationship is used to correct the overall pattern region and the overall template image, while the local mapping relationship is used to correct each local pattern region and its corresponding local template image.

[0030] In one specific embodiment, obtaining the target mapping relationship between the target pattern region and the target template image includes: randomly selecting multiple sets of feature point pairs from the target pattern region and the target template image, and determining the initial mapping relationship between the target pattern region and the target template image using the multiple sets of feature point pairs. This step is repeated to obtain multiple initial mapping relationships. The central tendency value of the multiple initial mapping relationships is obtained as the target mapping relationship. Wherein, the target pattern region is the overall pattern region, and the target mapping relationship is the overall mapping relationship; or, the target pattern region is a local pattern region, and the target mapping relationship is the local mapping relationship. The central tendency value is used to characterize the central position or general level of the multiple initial mapping relationships, and can be the mean, median, mode, etc., which are not limited here.

[0031] For example, based on the number of corresponding feature points in the target pattern region and the target template image, and using the RANSAC (Random Sample Consistency) method, multiple sets of feature point pairs are randomly selected from the target pattern region and the target template image. These multiple sets of feature point pairs are then used to determine the initial mapping relationship between the target pattern region and the target template image. Each set of feature point pairs contains the same number of feature points. This step is repeated to obtain multiple initial mapping relationships. The average mapping relationship of these multiple initial mapping relationships is then obtained as the target mapping relationship.

[0032] In one specific embodiment, before randomly selecting multiple sets of feature point pairs from the target pattern region and the target template image, or finding a local pattern region with fewer than a preset number of feature points as the region to be merged, feature points are filtered.

[0033] For example, based on the number of pixels in the target template image, the number of corresponding feature points in the target template image is obtained. For regions with a dense distribution of feature points (i.e., blocks in the target template image with a large number of feature points and rich information), a relatively large number of feature points can be retained. These feature points can be points located in the middle or at the edges of the region. For regions with a sparse distribution of feature points (i.e., regions in the target template image with a small number of feature points and poor information), the number of retained feature points can be reduced. Multiple pairs of feature points are randomly selected from the retained feature points in the target pattern region and the target template image. The specific feature point filtering operation can be adjusted according to actual needs and is not limited here.

[0034] Step S13: Based on the first pixel value difference between the aligned overall pattern region and the overall template image, determine the first defect location in the overall pattern region; and based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, determine the second defect location in each local pattern region.

[0035] The first pixel value difference and the second pixel value difference in this embodiment can be obtained using difference images, threshold segmentation, difference methods, template matching, decision trees, or deep learning. In one embodiment, the first defect location in the overall pattern region is determined based on the first pixel value difference between the aligned overall pattern region and the overall template image; or, the second defect location in each local pattern region is determined based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, including: obtaining a first difference image between the target template image and the target pattern region, wherein each pixel value in the first difference image represents the target pixel value difference between each pixel in the target pattern region and the corresponding pixel in the target template image. From the first difference image, a plurality of first target pixels with pixel values ​​greater than a preset difference threshold are found, wherein the plurality of first target pixels are used to define the first target defect location. Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, the target pixel value difference is the first pixel value difference, and the first target defect location is the first defect location; or, the target template image is the local template image, the target pattern area is the local pattern area, the target pixel value difference is the second pixel value difference, and the first target defect location is the second defect location.

[0036] In one specific implementation, a plurality of first target pixels are used to define the location of the first target defect. This can be achieved by treating the plurality of first target pixels as indicator points for the first target defect location, connecting these first target pixels, or expanding a small region centered on a single first target pixel, and using this region as the location of the first target defect. Alternatively, a plurality of first target pixels can be selected, deleting or ignoring those that are isolated, do not conform to defect characteristics, and using the region containing the selected first target pixels as the location of the first target defect. The specific method of using a plurality of first target pixels to define the location of the first target defect can be adjusted according to actual conditions and is not limited here.

[0037] The first target pixel can be determined by finding the relationship between the pixel value in the first difference image and a preset difference threshold, or by calculating the rate of change of pixel values ​​in adjacent pixels or a specific region in the first difference image, and identifying pixels with abnormal rates of change as the first target pixel. In one embodiment, finding a plurality of first target pixels whose pixel values ​​are greater than the preset difference threshold from the first difference image includes: obtaining a threshold image corresponding to the target template image from template information; setting the pixel values ​​of second target pixels in the first difference image to invalid values ​​to obtain a second difference image, wherein the pixel values ​​of the second target pixels are not greater than the pixel values ​​of the corresponding pixels in the threshold image; and finding a plurality of first target pixels whose pixel values ​​are greater than the preset difference threshold from the second difference image.

[0038] In another specific embodiment, obtaining a first difference image between the target template image and the target pattern region includes: obtaining a standard deviation image corresponding to the target template image from the template information; and obtaining a difference map between the target pattern region and the standard deviation image as the first difference image.

[0039] Before acquiring the first difference image, the target pattern region can be normalized, standardized, or linearly stretched to obtain an accurate difference map between the target pattern region and the standard deviation image. In one embodiment, before acquiring the first difference image between the target template image and the target pattern region, the method further includes: obtaining a normalization method from the template information. The target pattern region is normalized using the normalization method, and the normalized target pattern region is used to acquire the first difference image.

[0040] Taking the determination of the first target pixel as an example, a normalization method is obtained from the template information in the overall template image. This normalization method is then used to normalize the entire pattern region, resulting in an overall normalized image. The difference image between the standard deviation image in the template information and the overall normalized image is obtained as the first difference image. A threshold image is obtained from the template information, and pixels in the first difference image whose pixel values ​​exceed those in the threshold image are designated as the second target pixels. The pixel values ​​corresponding to the second target pixels in the first difference image are set to invalid values, resulting in the second difference image. Regions with significant differences in the second difference image are identified, and the pixels corresponding to these regions are designated as the first target pixels.

[0041] For example, a threshold image is derived from a target template image using a linear transformation. The linear transformation parameters (such as scaling or translation factors) determine the pixel value range of the threshold image and adjust its contrast and brightness. Pixels in the first difference image whose pixel values ​​are not greater than the corresponding pixel values ​​in the threshold image are set to 0 or a specific marker value. The second difference image contains pixels that are significantly different from the target template image. Pixels in the second difference image whose pixel values ​​are greater than a preset difference threshold are identified as the first target pixels.

[0042] Before determining the first defect location or the second defect location based on the difference in the first pixel value or the difference in the second pixel value, a deformation region with a large difference can be identified, and this deformation region can be directly output as the first defect location or the second defect location. In one embodiment, before determining the first defect location in the overall pattern region based on the difference in the first pixel value between the aligned overall pattern region and the overall template image, and before determining the second defect location in each local pattern region based on the difference in the second pixel value between each aligned local pattern region and the corresponding local template image, the method further includes: obtaining a target mapping relationship between the target pattern region and the target template image. In response to determining that the deformation degree of the target pattern region is greater than a preset degree threshold based on the target mapping relationship, a deformation region in the target pattern region whose deformation degree meets the deformation requirements is identified, and the deformation region is used as the second target defect location. The step of combining the first defect location and each second defect location to determine the third defect location of the target pattern in the image to be detected is then directly executed. Wherein, the target template image is the overall template image, the target pattern region is the overall pattern region, and the second target defect location is the first defect location, or the target template image is a local template image, the target pattern region is a local pattern region, and the second target defect location is the second defect location.

[0043] In another embodiment, before determining the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and before determining the second defect location in each local pattern region based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the method further includes: obtaining a target mapping relationship between the target pattern region and the target template image. In response to determining that the target pattern region is deformed based on the target mapping relationship and the degree of deformation is not greater than a preset degree threshold, the target pattern region is corrected using the target mapping relationship, wherein the corrected target pattern region is used for alignment and to determine the first target defect location in the target pattern region; before determining the third defect location of the target pattern in the image to be detected by combining the first defect location and each second defect location, the method further includes: mapping the first target defect location to the uncorrected target pattern region using the target mapping relationship. Wherein, the target template image is the overall template image, the target pattern region is the overall pattern region, and the first target defect location is the first defect location; or, the target template image is a local template image, the target pattern region is a local pattern region, and the first target defect location is the second defect location.

[0044] For details on how to obtain the target mapping relationship, please refer to the relevant description in step S12.

[0045] Taking the determination of the second defect location as an example, the local mapping relationship between each local pattern region and the local template image is obtained. Based on the relevant parameters in each local mapping relationship, it is determined whether each local pattern region is deformed and whether the degree of deformation is not greater than a threshold. For example, relevant parameters such as deformation gradient and deformation volume ratio are used for judgment. If a local pattern region is deformed and the degree of deformation is not greater than the threshold, the local pattern region is corrected using the corresponding local mapping relationship to obtain the corrected local pattern region. The corrected local pattern region is aligned with the corresponding local template image. Based on the difference in the second pixel value between the aligned local pattern region and the local template image, the location of the second defect in the local pattern region is determined. Using the corresponding local mapping relationship and the second difference image, the location of the second defect is mapped to the local pattern region before correction. If a local pattern region is deformed and the degree of deformation is greater than the threshold, the local pattern region is directly output as the location of the second defect.

[0046] Step S14: Combine the first defect location and each of the second defect locations to determine the third defect location of the target pattern in the image to be detected.

[0047] The third defect location of the target pattern in the image to be detected can be determined by combining the first defect location and the second defect locations through methods such as weighting, edge detection, morphological operations, and image registration. In one embodiment, determining the third defect location of the target pattern in the image to be detected by combining the first defect location and each of the second defect locations includes: weighting the first defect location and each of the second defect locations to obtain the third defect location.

[0048] In one specific embodiment, the second defect locations can be combined to obtain the local defect locations of the target pattern, wherein the local defect locations can be used to characterize the overall result of local detection. The local defect locations of the target pattern are weighted with the first defect locations to obtain the third defect locations.

[0049] In one embodiment, the first defect location, the second defect location, and the third defect location are all characterized using a mask image.

[0050] In another embodiment, this embodiment is implemented by a defect detection model, and the template information is learned by the defect detection model during training using defect-free pattern samples corresponding to the target pattern.

[0051] Combination Figure 2 To illustrate, Figure 2This is a schematic diagram of the training process of the defect detection model for the pattern defect detection method provided in this application. Defect-free pattern samples are collected, ideally encompassing the allowable defect error range. These samples are derived from one or more defect-free images; typically, multiple images are selected to mitigate differences caused by uneven illumination. The defect-free pattern samples undergo preprocessing, including noise reduction, filtering, and morphological transformations, to ensure consistent image quality. The standard deviation image, threshold image, and normalized image of the defect-free pattern samples are obtained, and the linear transformation parameters in the threshold image and the normalization method are set. Feature information is acquired, used for template alignment, comparison, filtering of fine-grained matching results, and mask generation. The mask can be a binary image, where defective regions are marked as 1 (or white) and non-defective regions are marked as 0 (or black). During training, the parameters and structure of the defect detection model can be continuously optimized to improve the accuracy and robustness of defect detection. Through a large number of defect-free pattern samples and repeated training iterations, the defect detection model can learn to capture feature information in the image to be detected, achieving accurate detection of pattern defects. Based on the template training method described above, corresponding global and local templates are trained, and corresponding global or local template images can be obtained from the global or local templates. Both the global and local template images include two parts: a template image and feature information. The template image includes a standard deviation image, a threshold image, and a normalized image calculated using defect-free images. The feature information is used for template alignment, comparison, filtering of fine-grained matching results, and mask generation.

[0052] This embodiment aligns the overall pattern region with the overall template image, and aligns each local pattern region with its corresponding local template image. Based on the first pixel value difference between the aligned overall pattern region and the overall template image, the first defect location in the overall pattern region is determined. Similarly, based on the second pixel value difference between each aligned local pattern region and its corresponding local template image, the second defect location in each local pattern region is determined. Combining the first and second defect locations, the third defect location of the target pattern in the image to be detected is determined. This application can identify the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image. For each local pattern region, the second pixel value difference between it and its corresponding local template image is also calculated to determine the second defect location in each local pattern region. By comprehensively analyzing the first and second defect locations and integrating defect information from different dimensions, the third defect location of the target pattern in the image to be detected is obtained. When dealing with complex situations such as distortion or irregular deformation in the image, this application employs a block-based detection method to achieve in-depth analysis of the details of distortion or irregular deformation. Simultaneously, a whole-image detection method is used to perform an overall analysis of the image's condition. This combination of whole-image detection and block detection enables this application to more comprehensively capture and accurately locate defects in the target pattern, thereby improving the accuracy of defect detection.

[0053] Please see Figure 3 , Figure 3 This is a partial flowchart illustrating a specific embodiment of the pattern defect detection method provided in this application. Figure 3 As shown, the pattern defect detection method provided in this embodiment may include the following steps:

[0054] Step S31: Obtain the overall pattern region of the target pattern in the image to be detected. Step S32: Based on the feature information of the target pattern, divide the target pattern into blocks to obtain several local pattern regions. Step S33: Align the overall pattern region with the overall template image, and align each local pattern region with its corresponding local template image. Step S34: Extract the overall feature information from the overall pattern region, and extract the local feature information from each local pattern region. Step S35: Match and compare the overall feature information with the template image in the overall template image, and match and compare each local feature information with the template image in its corresponding local template image. Step S36: Determine if the matching and comparison is successful. If the matching and comparison is successful, proceed to step S37: Combine the first defect location and the second defect location to output the third defect location. If the matching and comparison fails, proceed to step S38: Determine that the target pattern defect is too large and invalid.

[0055] Please see Figure 4 , Figure 4 This is a schematic flowchart of an embodiment of the pattern defect detection device of this application. The pattern defect detection device 400 includes a pattern acquisition module 410, an alignment module 420, a defect location acquisition module 430, and a defect location synthesis module 440. The pattern acquisition module 410 is used to acquire the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, and to acquire the overall template image and several local template images from the template information corresponding to the target pattern. Each local pattern region corresponds to a different local region of the target pattern. The overall template image and the local template images are the overall image and local image of the target pattern in a defect-free state, respectively, and each local pattern region corresponds one-to-one with each local template image. The alignment module 420 is used to align the overall pattern region with the overall template image, and to align each local pattern region with its corresponding local template image. The defect location acquisition module 430 is used to determine the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and to determine the second defect location in each local pattern region based on the second pixel value difference between the aligned local pattern region and its corresponding local template image. The defect location integration module 440 is used to integrate the first defect location and each of the second defect locations to determine the third defect location of the target pattern in the image to be detected.

[0056] In some embodiments, the defect location acquisition module 430 determines the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, or determines the second defect location in each local pattern region based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, including: acquiring a first difference image between the target template image and the target pattern region, wherein each pixel value in the first difference image represents the target pixel value difference between each pixel in the target pattern region and the corresponding pixel in the target template image. From the first difference image, a plurality of first target pixels with pixel values ​​greater than a preset difference threshold are identified, wherein the plurality of first target pixels are used to define the first target defect location. Wherein, the target template image is the overall template image, the target pattern region is the overall pattern region, the target pixel value difference is the first pixel value difference, and the first target defect location is the first defect location; or, the target template image is a local template image, the target pattern region is a local pattern region, the target pixel value difference is the second pixel value difference, and the first target defect location is the second defect location.

[0057] In some embodiments, the defect location acquisition module 430, when performing the process of finding a plurality of first target pixels whose pixel values ​​are greater than a preset difference threshold from a first difference image, includes: obtaining a threshold image corresponding to a target template image from template information; setting the pixel values ​​of second target pixels in the first difference image to invalid values ​​to obtain a second difference image, wherein the pixel values ​​of the second target pixels are not greater than the pixel values ​​of the corresponding pixels in the threshold image; and finding a plurality of first target pixels whose pixel values ​​are greater than the preset difference threshold from the second difference image.

[0058] In some embodiments, the defect location acquisition module 430, when performing the acquisition of a first difference image between a target template image and a target pattern region, includes: acquiring a standard deviation image corresponding to the target template image from template information; and acquiring a difference map between the target pattern region and the standard deviation image as the first difference image.

[0059] In some embodiments, before the defect location acquisition module 430 acquires the first difference image between the target template image and the target pattern region, the method further includes: acquiring a normalization method from the template information. The target pattern region is normalized using the normalization method, wherein the normalized target pattern region is used to acquire the first difference image.

[0060] In some embodiments, the pattern defect detection device 400 further includes a deformation degree detection module. Before determining the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and before determining the second defect location in each local pattern region based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the deformation degree detection module includes: acquiring a target mapping relationship between the target pattern region and the target template image. In response to determining that the deformation degree of the target pattern region is greater than a preset degree threshold based on the target mapping relationship, a deformed region in the target pattern region whose deformation degree meets the deformation requirements is determined, and the deformed region is used as the second target defect location. Then, the step of combining the first defect location and each second defect location to determine the third defect location of the target pattern in the image to be detected is directly executed. Wherein, the target template image is the overall template image, the target pattern region is the overall pattern region, and the second target defect location is the first defect location; or, the target template image is a local template image, the target pattern region is a local pattern region, and the second target defect location is the second defect location.

[0061] In some embodiments, the pattern defect detection device 400 further includes a deformation degree detection module. Before determining the first defect position in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and before determining the second defect position in each local pattern region based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the deformation degree detection module includes: acquiring a target mapping relationship between the target pattern region and the target template image. In response to determining that the target pattern region is deformed based on the target mapping relationship and the deformation degree is not greater than a preset degree threshold, the target pattern region is corrected using the target mapping relationship, wherein the corrected target pattern region is used for alignment and to determine the first target defect position in the target pattern region. Before determining the third defect position of the target pattern in the image to be detected by combining the first defect position and each second defect position, the module further includes: mapping the first target defect position to the target pattern region before correction using the target mapping relationship. Wherein, the target template image is the overall template image, the target pattern region is the overall pattern region, and the first target defect position is the first defect position; or, the target template image is a local template image, the target pattern region is a local pattern region, and the first target defect position is the second defect position.

[0062] In some embodiments, the deformation degree detection module, in performing the acquisition of the target mapping relationship between the target pattern region and the target template image, includes: randomly selecting multiple sets of feature point pairs from the target pattern region and the target template image, and determining an initial mapping relationship between the target pattern region and the target template image using the multiple sets of feature point pairs. This step is repeated to obtain multiple initial mapping relationships. The central tendency representation value of the multiple initial mapping relationships is obtained as the target mapping relationship.

[0063] In some embodiments, the pattern defect detection device 400 further includes a segmentation module. Before aligning the overall pattern area with the overall template image and aligning each local pattern area with its corresponding local template image, the segmentation module includes: identifying local pattern areas with fewer than a preset number of feature points as areas to be merged; merging the areas to be merged with adjacent local pattern areas to form new local pattern areas; and merging the local template images corresponding to the areas to be merged and the local template images corresponding to the adjacent local pattern areas to form new local template images.

[0064] In some embodiments, before the deformation degree detection module or the block segmentation module performs the random selection of multiple sets of feature point pairs from the target pattern region and the target template image, or finds a local pattern region with fewer than a preset number of feature points as the region to be merged, the module further includes: filtering the feature points in the target pattern region and the target template region based on the number of pixels in the target template image.

[0065] In some embodiments, the alignment module 420 performs the following steps when aligning the overall pattern region with the overall template image, or aligning each local pattern region with its corresponding local template image: obtaining fine-grained alignment information from the template information; and aligning the overall pattern region with the overall template image or the local pattern region with its corresponding local template image according to the fine-grained alignment information.

[0066] In some embodiments, the defect location synthesis module 440 determines the third defect location of the target pattern in the image to be detected by synthesizing the first defect location and each of the second defect locations, including weighting the first defect location and each of the second defect locations to obtain the third defect location.

[0067] In some embodiments, the pattern acquisition module 410 performs the acquisition of the overall pattern region by: using a target detection model to perform target detection on the image to be detected to obtain the overall pattern region; or, using an overall template image corresponding to the target pattern to perform template matching on the image to be detected to obtain the overall pattern region. In response to a mismatch between the size of the overall pattern region and the overall template image, the size of the overall pattern region is adjusted to be consistent with the overall template image.

[0068] In some embodiments, the pattern acquisition module 410 performs the acquisition of several local pattern regions by dividing the overall pattern region into blocks according to the method of overlapping sliding windows to obtain several local pattern regions, wherein the window of the overlapping sliding window is the same size as the local template image, and the overlap ratio of the overlapping sliding window is preset or input by the user.

[0069] In some embodiments, before the pattern acquisition module 410 performs the acquisition of the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, it further includes: preprocessing the image to be detected, wherein the preprocessing includes at least one of the following: noise reduction, filtering and morphological transformation.

[0070] In some embodiments, the pattern defect detection method is implemented by a defect detection model, and the template information is learned by the defect detection model during training using defect-free pattern samples corresponding to the target pattern.

[0071] In some embodiments, the first defect location, the second defect location, and the third defect location are all characterized using a mask image.

[0072] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above-described embodiments of the pattern defect detection method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0073] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described embodiments of the pattern defect detection method. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0074] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described embodiments of the pattern defect detection method.

[0075] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0076] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for detecting pattern defects, characterized in that, The method includes: The overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected are obtained, and the overall template image and several local template images are obtained from the template information corresponding to the target pattern. Each of the local pattern regions corresponds to a different local region of the target pattern. The overall template image and the local template image are the overall image and local image of the target pattern in a defect-free state, respectively. Each of the local pattern regions corresponds one-to-one with each of the local template images. Align the overall pattern area with the overall template image, and align each of the local pattern areas with the corresponding local template image; Based on the first pixel value difference between the aligned overall pattern region and the overall template image, the first defect position in the overall pattern region is determined, and based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the second defect position in each local pattern region is determined. By combining the first defect location and each of the second defect locations, the third defect location of the target pattern in the image to be detected is determined.

2. The method according to claim 1, characterized in that, The step of determining the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, or the step of determining the second defect location in each of the local pattern regions based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, includes: A first difference image is obtained between a target template image and a target pattern region, wherein each pixel value in the first difference image represents the difference in target pixel values ​​between each pixel in the target pattern region and the corresponding pixel in the target template image; From the first difference image, find a number of first target pixels whose pixel values ​​are greater than a preset difference threshold, wherein the number of first target pixels are used to define the location of the first target defect; Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, the target pixel value difference is the first pixel value difference, and the first target defect position is the first defect position; or, the target template image is the partial template image, the target pattern area is the partial pattern area, the target pixel value difference is the second pixel value difference, and the first target defect position is the second defect position.

3. The method according to claim 2, characterized in that, The step of finding a plurality of first target pixels whose pixel values ​​are greater than a preset difference threshold from the first difference image includes: From the template information, obtain the threshold image corresponding to the target template image; The pixel value of the second target pixel in the first difference image is set to an invalid value to obtain a second difference image, wherein the pixel value of the second target pixel is not greater than the pixel value of the corresponding pixel in the threshold image; From the second difference image, find the plurality of first target pixels whose pixel values ​​are greater than the preset difference threshold.

4. The method according to claim 2, characterized in that, The step of obtaining the first difference image between the target template image and the target pattern region includes: Obtain the standard deviation image corresponding to the target template image from the template information; Obtain the difference image between the target pattern region and the standard deviation image, and use it as the first difference image; And / or, before acquiring the first difference image between the target template image and the target pattern region, the method further includes: Obtain the normalization method from the template information; The target pattern region is normalized using the normalization method described above, and the normalized target pattern region is used to obtain the first difference image.

5. The method according to claim 1, characterized in that, Before determining the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and before determining the second defect location in each of the local pattern regions based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the method further includes: Obtain the target mapping relationship between the target pattern region and the target template image; In response to determining that the deformation degree of the target pattern region is greater than a preset degree threshold based on the target mapping relationship, a deformation region in the target pattern region whose deformation degree meets the deformation requirements is determined, the deformation region is taken as the second target defect location, and the step of combining the first defect location and each of the second defect locations to determine the third defect location of the target pattern in the image to be detected is directly executed; Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, and the second target defect location is the first defect location; or, the target template image is the partial template image, the target pattern area is the partial pattern area, and the second target defect location is the second defect location.

6. The method according to claim 1, characterized in that, Before determining the first defect location in the overall pattern region based on the first pixel value difference between the aligned overall pattern region and the overall template image, and before determining the second defect location in each of the local pattern regions based on the second pixel value difference between each aligned local pattern region and the corresponding local template image, the method further includes: Obtain the target mapping relationship between the target pattern region and the target template image; In response to determining, based on the target mapping relationship, that the target pattern region is deformed and the degree of deformation is not greater than a preset degree threshold, the target pattern region is corrected using the target mapping relationship, wherein the corrected target pattern region is used for the alignment and the determination of the first target defect position in the target pattern region; Before determining the third defect location of the target pattern in the image to be detected by combining the first defect location and each of the second defect locations, the method further includes: Using the target mapping relationship, the location of the first target defect is mapped to the target pattern area before correction; Wherein, the target template image is the overall template image, the target pattern area is the overall pattern area, and the first target defect location is the first defect location; or, the target template image is the partial template image, the target pattern area is the partial pattern area, and the first target defect location is the second defect location.

7. The method according to claim 5 or 6, characterized in that, The process of obtaining the target mapping relationship between the target pattern region and the target template image includes: Randomly select multiple sets of feature point pairs from the target pattern region and the target template image, and determine the initial mapping relationship between the target pattern region and the target template image using the multiple sets of feature point pairs; repeat this step to obtain multiple initial mapping relationships; Obtain the central tendency representation values ​​of multiple initial mapping relationships as the target mapping relationship.

8. The method according to claim 7, characterized in that, Before randomly selecting multiple sets of feature point pairs from the target pattern region and the target template image, the method further includes: Based on the number of pixels in the target template image, feature points in the target pattern region and the target template region are filtered.

9. The method according to claim 1, characterized in that, Before aligning the overall pattern region with the overall template image, and before aligning each of the local pattern regions with the corresponding local template image, the method further includes: The local pattern regions with fewer than a preset number of feature points are identified as regions to be merged, and these regions are merged with adjacent local pattern regions to form new local pattern regions. The local template image corresponding to the region to be merged and the local template image corresponding to the adjacent local pattern region are merged to form a new local template image.

10. The method according to claim 1, characterized in that, The step of aligning the overall pattern region with the overall template image, or the step of aligning each of the local pattern regions with the corresponding local template image, includes: Obtain fine-grained alignment information from the template information; According to the fine-grained alignment information, the overall pattern area is aligned with the overall template image, or the local pattern area is aligned with the corresponding local template image; And / or, determining the third defect location of the target pattern in the image to be detected by combining the first defect location and each of the second defect locations includes: The third defect position is obtained by weighting the first defect position and each of the second defect positions.

11. The method according to claim 1, characterized in that, Obtaining the overall pattern area includes: The overall pattern region is obtained by performing target detection on the image to be detected using a target detection model; or, the overall pattern region is obtained by performing template matching on the image to be detected using the overall template image corresponding to the target pattern. In response to the inconsistency between the size of the overall pattern area and the overall template image, the size of the overall pattern area is adjusted to match that of the overall template image. And / or, acquiring the plurality of local pattern regions, including: The overall pattern area is divided into blocks according to the method of overlapping sliding windows to obtain the several local pattern areas. The window of the overlapping sliding window is the same size as the local template image, and the overlap ratio of the overlapping sliding window is preset or input by the user. And / or, before acquiring the overall pattern region and several local pattern regions corresponding to the target pattern in the image to be detected, the method further includes: The image to be detected is preprocessed, wherein the preprocessing includes at least one of the following: noise reduction, filtering, and morphological transformation.

12. The method according to claim 1, characterized in that, The method is implemented by a defect detection model, and the template information is learned by the defect detection model during training using defect-free pattern samples corresponding to the target pattern. And / or, the first defect location, the second defect location, and the third defect location are all characterized using a mask image.

13. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to implement the pattern defect detection method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the pattern defect detection method as described in any one of claims 1-12.

Images