Defect detection method, device and equipment of integrated circuit image and storage medium

Abstract

The application relates to a defect detection method, device and equipment of an integrated circuit image and a storage medium. The method comprises the following steps: acquiring original feature maps of a plurality of defect images and candidate detection feature maps of candidate regions in a to-be-detected image; for each defect image, performing granularity extraction on the original feature maps of the defect image to obtain a plurality of target granularity feature maps; performing cross-granularity fusion on a feature map group according to a preset granularity fusion scheme to obtain a fused granularity feature set; performing feature reconstruction on the same defect type according to the fused granularity feature set corresponding to the defect images of the same defect type to obtain a feature prototype corresponding to the same defect type; and determining a target defect type of the candidate detection feature maps based on the feature prototypes corresponding to the defect types of the plurality of defect images. The method can capture rich feature information to enhance the recognition and detection capability of defects, and improve the accuracy and reliability of defect detection.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to a method, apparatus, device, and storage medium for detecting defects in integrated circuit images. Background Technology

[0002] Defect detection is crucial for ensuring product quality control. Integrated circuit defect detection and identification spans different lifecycle stages, including manufacturing, packaging, and final product testing, and is a vital link in ensuring product quality. During the production and R&D of integrated circuits, multiple inspections are required. However, due to the complexity of integrated circuit manufacturing processes and the continuous shrinking of feature sizes, the occurrence of defects is difficult to avoid.

[0003] In related technologies, deep learning-based methods are commonly used. This technique builds models using a large amount of training data, and then uses these models to learn features in integrated circuit images to detect defects. It can handle complex defect situations to a certain extent.

[0004] However, in related technologies, due to the variability of defects, the numerous integrated circuit models and defect types, and the varying sizes of defects, the false negative rate is extremely high when directly using deep learning models to detect packaging or internal defects, significantly reducing detection accuracy. Furthermore, defect categories exhibit fine-grained image characteristics, with significant differences even within the same defect category, and high similarity between defects and good products, greatly increasing the difficulty of defect category identification. Additionally, because there are numerous good product images but few defects, defects that were not observed during training may appear during production, leading to their inability to be identified during detection. Therefore, improving the accuracy of defect detection has become an urgent problem to be solved in this field. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for detecting defects in integrated circuit images that can capture rich feature information to enhance the ability to identify and detect defects, thereby reducing the possibility of missed detection and improving the accuracy and reliability of defect detection.

[0006] In a first aspect, this application provides a method for defect detection in integrated circuit images, comprising:

[0007] Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected;

[0008] For each defect image, the original feature map of the defect image is extracted at multiple granularities to obtain multiple target granularity feature maps; the target granularity feature maps include feature blocks with different spatial coordinates and different granularity sizes;

[0009] The feature map group is fused across granularities according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps;

[0010] Based on the fusion granularity feature set corresponding to the defect images of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type.

[0011] Based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined.

[0012] In one embodiment, the original feature map of the defect image is subjected to multi-granularity extraction to obtain multiple target-granularity feature maps, including:

[0013] Determine the particle size distribution scheme for each particle size; the particle size distribution scheme includes the particle size division dimensions;

[0014] The original feature map is divided into granularities according to each granularity partitioning scheme to obtain the initial feature correspondence; the initial feature correspondence includes the correspondence between different granularity feature blocks and the initial spatial coordinates;

[0015] The initial feature correspondence is processed sequentially to obtain the target feature correspondence; the target feature correspondence includes the correspondence between feature blocks of different granularities and the reconstructed spatial coordinates.

[0016] In one embodiment, the preset granularity fusion scheme includes a preset exchange size. Cross-granularity fusion is performed on all target granularity feature maps according to the preset granularity fusion scheme to obtain a fused granularity feature set, including:

[0017] For each feature map group, based on a preset exchange size, exchange feature blocks are selected from two target granularity feature maps in the feature map group; the size of the exchange feature blocks matches the preset exchange size.

[0018] The exchanged feature blocks of the two target granularity feature maps in the feature map group are replaced to obtain the fused feature map group corresponding to the feature map group;

[0019] The fusion granularity feature set includes the fusion feature map set of each feature map set.

[0020] In one embodiment, based on a preset exchange size, an exchange feature block is selected from two target granularity feature maps in a feature map group, including:

[0021] Determine the starting point of the first region of the first granularity feature map and the starting point of the second region of the second granularity feature map; the first granularity feature map is a target granularity feature map of the feature map group, and the second granularity feature map is another target granularity feature map of the feature map group;

[0022] The first exchange region is determined based on the preset exchange size and the starting point of the first region, and the first granularity feature block located within the target exchange region is selected from the first granularity feature map;

[0023] The second exchange region is determined based on the preset exchange size and the starting point of the second region, and the second granularity feature blocks located within the second exchange region are selected from the second granularity feature map.

[0024] The exchanged feature blocks include the first-granularity feature block and the second-granularity feature block of the feature map group.

[0025] In one embodiment, the exchanged feature blocks of two target-granularity feature maps in the feature map group are permuted to obtain a fused feature map group corresponding to the feature map group, including:

[0026] A first mask for generating a first granularity feature map based on a first exchange region, and a second mask for generating a second granularity feature map based on a second exchange region;

[0027] Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the first granularity feature map to obtain the first fused feature map;

[0028] Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the second granularity feature map to obtain the second fused feature map;

[0029] The fused feature map group includes a first fused feature map. Second fusion feature map The specific calculation method is as follows:

[0030]

[0031]

[0032]

[0033]

[0034] In the above calculation method, These represent the first granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, These represent the second granularity feature maps respectively. Position coordinates in the horizontal and vertical directions This is the first exchange area. This is the second exchange area.

[0035] In one embodiment, the fusion granularity feature set includes fused feature map sets of each feature map group; based on the fusion granularity feature set corresponding to the defect image of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type, including:

[0036] For each defect type, determine the fused feature map in all fused feature map groups corresponding to the defect type;

[0037] Based on the ridge regression model, feature reconstruction is performed on the fused feature maps in all fused feature map groups of the same defect to obtain the feature prototype corresponding to the defect type.

[0038] In one embodiment, the target defect type of the candidate detection feature map is determined based on the feature prototypes corresponding to each defect type in multiple defect images, including:

[0039] The candidate detection feature map is transformed to obtain candidate detection features;

[0040] For each defect type, the similarity between the candidate detection features and the feature prototype is calculated to obtain the feature similarity.

[0041] The target defect type is determined as the defect type corresponding to the highest feature similarity.

[0042] Secondly, this application also provides a defect detection device for integrated circuit images, comprising:

[0043] The data acquisition module is used to acquire the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected;

[0044] The multi-granularity partitioning module is used to extract multiple target granularity feature maps from the original feature maps of each defect image. The target granularity feature maps include the correspondence between different spatial coordinates and different granularity feature blocks.

[0045] The cross-granularity fusion module is used to perform cross-granularity fusion on feature map groups according to a preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps;

[0046] The feature prototype construction module is used to perform feature reconstruction on the same defect type based on the fusion granularity feature set corresponding to the defect image of the same defect type, and obtain the feature prototype corresponding to the same defect type.

[0047] The defect determination module is used to determine the target defect type of the candidate detection feature map based on the feature prototypes corresponding to each defect type in multiple defect images.

[0048] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0049] Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected;

[0050] For each defect image, the original feature map of the defect image is extracted at multiple granularities to obtain multiple target granularity feature maps; the target granularity feature maps include the correspondence between different spatial coordinates and different granularity feature blocks;

[0051] The feature map group is fused across granularities according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps;

[0052] Based on the fusion granularity feature set corresponding to the defect images of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type.

[0053] Based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined.

[0054] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0055] Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected;

[0056] For each defect image, the original feature map of the defect image is extracted at the granular level to obtain multiple target granular feature maps; the target granular feature maps include the correspondence between different spatial coordinates and different granular feature blocks;

[0057] The feature map group is fused across granularities according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps;

[0058] Based on the fusion granularity feature set corresponding to the defect images of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type.

[0059] Based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined.

[0060] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0061] Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected;

[0062] For each defect image, the original feature map of the defect image is extracted at the granular level to obtain multiple target granular feature maps; the target granular feature maps include the correspondence between different spatial coordinates and different granular feature blocks;

[0063] The feature map group is fused across granularities according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps;

[0064] Based on the fusion granularity feature set corresponding to the defect images of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type.

[0065] Based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined.

[0066] The aforementioned integrated circuit image defect detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product extract multiple target granularity feature maps from the original feature maps of multiple acquired defect images, enabling the mining of defect features at different granular levels and capturing rich feature information. Then, cross-granularity fusion is performed on the extracted target granularity feature maps to integrate features of different granularities, thereby enhancing the expressive power of the features and facilitating improved defect recognition. Next, features from different defect images belonging to the same defect type are integrated to form a typical feature representation of that defect type, i.e., a feature prototype. Then, the candidate detection feature map of the candidate region in the acquired image to be detected is transformed into a feature space in the same space as the feature prototype. By calculating the similarity between the candidate detection feature map and the feature prototype, the defect type of the candidate region in the image to be detected can be determined by comparing the similarity, thus improving the accuracy and comprehensiveness of defect detection. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1This is an application environment diagram of a defect detection method for integrated circuit images in one embodiment;

[0069] Figure 2 This is a flowchart illustrating a defect detection method for integrated circuit images in one embodiment;

[0070] Figure 3 This is a schematic diagram of the process for obtaining a target granularity feature map in one embodiment;

[0071] Figure 4 This is a schematic diagram of the process for obtaining a fusion granularity feature set in one embodiment;

[0072] Figure 5 This is a flowchart illustrating the process of filtering out exchange feature blocks in one embodiment;

[0073] Figure 6 This is a schematic diagram of the process for obtaining a fused feature map group in one embodiment;

[0074] Figure 7 This is a schematic diagram of the process for obtaining a feature prototype in one embodiment;

[0075] Figure 8 This is a flowchart illustrating the process of determining the target defect type of a candidate detection feature map in one embodiment.

[0076] Figure 9 This is a structural block diagram of a defect detection device for integrated circuit images in one embodiment;

[0077] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. In the description of embodiments of this application, technical terms such as "first," "second," etc., are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary or secondary relationship of the indicated technical features. In the description of embodiments of this application, "a plurality of" means two or more, unless otherwise explicitly defined.

[0080] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0081] In related technologies, integrated circuits (ICs) exhibit a wide variety of models and defect types, with varying sizes and dimensions, some being extremely small and elongated. Layout comparison methods are unsuitable for high-variety, low-batch production scenarios. Directly applying deep learning-based models to detect defects in IC packages or internal components results in a very high false negative rate and a significant drop in accuracy because the models struggle to capture the features of minute targets. Furthermore, defect categories exhibit fine-grained image characteristics. The same defect category shows significant differences across different locations or samples, while defects often share a high degree of similarity with good products. For example, void defects vary in shape, including ring-shaped, circular, and irregular shapes. Additionally, lighter-colored areas within an IC may represent good design features or void defects, making defect classification extremely difficult.

[0082] Based on this, embodiments of this application provide a method, apparatus, device, and storage medium for defect detection of integrated circuit images, which can effectively capture rich feature information, enhance the ability to identify and detect defects by means of feature information, thereby reducing the possibility of missed detection and significantly improving the accuracy and reliability of defect detection.

[0083] The integrated circuit image defect detection method provided in this application embodiment can be executed by a single electronic device with computing power, or by a system composed of multiple electronic devices. For example, the integrated circuit image defect detection method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located on a cloud or other network server. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0084] In one embodiment, such as Figure 2 As shown, a defect detection method for integrated circuit images is provided, which includes the following steps:

[0085] Step 202: Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected.

[0086] In such Figure 1 In the application environment shown, terminal 102 is equipped with image detection software for defect detection of various images. In practice, the image detection software is used to acquire images of integrated circuits, and the acquired images are processed to facilitate comparison and analysis with known defect images.

[0087] For example, defective product images detected on the production line or typical defect images are obtained from relevant databases, and the obtained images are processed to obtain defect images of various defect types. Then, feature extraction is performed on the defect images to obtain the original feature map of the defect image, wherein the original feature map includes, but is not limited to, the shape, texture and color of the defect.

[0088] In some possible implementations, feature extraction employs traditional algorithms. Taking HOG (Histogram of Oriented Gradients) as an example, the image is normalized, then divided into multiple image units. Within each unit, a gradient orientation histogram is calculated to collect information on the direction and magnitude of the gradient. Adjacent image units are then combined into blocks, and the histograms within these blocks are normalized to obtain the block's feature vector. Finally, the feature vectors from all blocks are concatenated to form the final feature map.

[0089] In other possible embodiments, feature extraction employs deep learning. Taking a convolutional neural network (CNN) as an example, the defective image is input into the CNN and first passes through a convolutional layer. Multiple convolutional kernels within the convolutional layer slide across the image, multiplying corresponding elements and summing them to obtain the values ​​on the feature map. Different kernels extract different features, such as edges and textures. After convolution, a non-linear transformation is performed using activation functions such as ReLU. Then, pooling layers are connected for downsampling to reduce the feature map size. As the network deepens, the number of feature maps increases, their size decreases, and their abstraction level improves, ultimately resulting in feature maps containing rich semantic information.

[0090] Simultaneously, the image to be detected needs to be processed. The target detection algorithm is used to determine the regions in the image to be detected that may contain defects, i.e., candidate regions. Then, by extracting the original feature map of the defect image, the candidate regions are used to extract features to obtain the candidate detection feature map of the candidate regions.

[0091] Step 204: For each defect image, perform multi-granularity extraction on the original feature map of the defect image to obtain multiple target granularity feature maps; the target granularity feature maps include feature blocks with different spatial coordinates and different granularity sizes.

[0092] Since the obtained original feature map contains various feature information of defects, but the feature information is relatively complex and general, in order to improve the identification of the defect type corresponding to the candidate detection feature map using the original feature map, it is necessary to further mine the feature information in the original feature map.

[0093] For example, for the original feature map of each defect image, a granular extraction algorithm is needed to divide the original feature map according to different granularity standards, thereby obtaining multiple target granularity feature maps. Each target granularity feature map contains multiple granular feature blocks of the same granularity, and the specific location of each granular feature block within the target granularity feature map, i.e., its spatial coordinates, can be clearly defined. The spatial coordinates specify the exact location of the feature in the image, while the granular feature block contains the feature information at that location. For example, when detecting defects in integrated circuits, a granular feature block corresponding to a certain spatial coordinate might represent an anomaly in a solder joint. By analyzing different target granularity feature maps, we can gain a comprehensive understanding of the morphology, size, and distribution of defects from macroscopic to microscopic perspectives.

[0094] In some possible embodiments, a multi-granularity generator consisting of multiple sub-generators is used to divide the original feature map into blocks. Different sub-generators divide the original feature map into feature blocks of different sizes. That is, through the collaborative work of multiple sub-generators, features of different granularities are extracted from the original feature map to generate multiple target granularity feature maps.

[0095] In other possible embodiments, a multi-scale convolution approach is employed. Convolution operations are performed on the original feature map using kernels of different sizes. For example, small kernels can capture detailed information in the feature map, such as tiny cracks and imperfections; large kernels can extract macroscopic features, such as large damaged areas. By performing convolutions at different scales, multiple feature maps of different scales can be obtained. Further processing and integration of these feature maps at different scales yields multiple target-granularity feature maps.

[0096] Step 206: Perform cross-granularity fusion on the feature map group according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps.

[0097] Different target granularity feature maps reflect the feature information of defect images at their respective granularity levels, but features at a single granularity generally have limitations. To avoid these limitations reducing the accuracy of defect identification, target granularity feature maps of different granularities can be fused to integrate the feature information from these maps, thereby improving the accuracy of defect identification.

[0098] For example, when performing cross-granularity fusion, it is necessary to group multiple target granularity feature maps, that is, combine any two target granularity feature maps to obtain multiple feature map groups. For each feature map group, a fusion operation is performed according to a preset granularity fusion scheme. For example, the preset granularity fusion scheme can adopt a weighted summation method, assigning corresponding weights according to the importance of different granularity features, and then weighted summing the elements at corresponding positions of the two feature maps; alternatively, it can select regions of the same size for each of the two target granularity feature maps in the feature map group, and swap the granularity feature blocks within the selected regions, so that each target granularity feature map has the feature information of the other target granularity feature map. Then, after fusion, a fused feature map group containing multiple feature map groups can be obtained, that is, a fused granularity feature set.

[0099] Step 208: Based on the fusion granularity feature set corresponding to the defect images of the same defect type, perform feature reconstruction for the same defect type to obtain the feature prototype corresponding to the same defect type.

[0100] The fusion-granularity feature set is obtained by merging any two target granularity feature maps from multiple sources. The fused feature map set contains rich and comprehensive defect feature information. However, this feature information is still scattered across the individual fused feature maps. Therefore, it is necessary to integrate the feature information from the fused feature maps within the fused feature set.

[0101] For example, the fusion granularity feature set corresponding to the defect images of the same defect type is organized to obtain the fusion feature map in the fusion feature map group corresponding to each defect image. Then, by analyzing and integrating all the fusion feature maps belonging to the same defect type, information that can represent the essential features of the defect type is extracted, and the information representing the essential features of the defect type is integrated into a feature prototype vector.

[0102] In some possible embodiments, a ridge regression model is introduced to process all fused feature maps of the same defect type. Specifically, the ridge regression method adds a regularization term to ordinary linear regression to avoid overfitting. All fused feature maps of the same defect type are input into the ridge regression model, which then learns from them. By continuously adjusting the model parameters of the ridge regression model, it is possible to accurately describe the feature patterns of the defect type, thereby obtaining the feature prototypes corresponding to the same defect type. Step 210: Based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined.

[0103] The feature prototypes corresponding to each defect type can accurately summarize the core features of each defect type. At this point, the candidate detection feature map of the image to be detected can be compared with the feature prototypes corresponding to each defect type to determine which type the defect of the image to be detected belongs to.

[0104] For example, the candidate detection feature map is converted into a feature, and then the similarity between the feature and the feature prototype of each defect type is calculated. The similarity calculation method includes, but is not limited to, Euclidean distance and cosine similarity. The smaller the Euclidean distance, the closer the two features are; the closer the cosine similarity is to 1, the more similar the two features are.

[0105] Then, based on the calculated similarity results, the defect type corresponding to the feature prototype of the defect type with the highest similarity is selected as the target defect type of the candidate detection feature map. If all similarities are below a certain preset threshold, it can be determined that the candidate detection feature map does not belong to an existing defect type, and may be a new defect or a normal image.

[0106] In the aforementioned defect detection method for integrated circuit images, multiple target granularity feature maps are obtained by extracting the original feature maps of multiple defect images at different granularities. This allows for the mining of defect features at different granularity levels and the capture of rich feature information. Then, cross-granularity fusion is performed on the extracted target granularity feature maps to integrate features of different granularities, thereby enhancing the expressive power of the features and facilitating improved defect identification. Next, features from different defect images belonging to the same defect type are integrated to form a typical feature representation of that defect type, i.e., a feature prototype. Then, the candidate detection feature map of the candidate region in the image to be detected is transformed into a feature space in the same space as the feature prototype. By calculating the similarity between the candidate detection feature map and the feature prototype, the defect type of the candidate region in the image to be detected can be determined by comparing the similarity, thus improving the accuracy and comprehensiveness of defect detection.

[0107] In practical applications, when analyzing defect images to determine the target defect type in candidate regions of an image to be detected, it is necessary to first obtain the original feature map and candidate detection feature map. Then, granular extraction and cross-granularity fusion are performed on the original feature map to obtain a fused granularity feature set. Subsequently, a feature prototype is constructed based on the fused granularity feature set. Finally, the candidate detection feature map of the candidate region is compared with the feature prototype to determine the target defect type. This process aims to more accurately identify defect types at the feature level. The accuracy and precision of the granular extraction operation directly affect subsequent feature fusion, prototype construction, and defect type determination. Therefore, further exploration is needed to determine the specific implementation methods for granular extraction of the original feature map of the defect image to obtain multiple target granularity feature maps.

[0108] In one embodiment, such as Figure 3 As shown, multi-granularity extraction is performed on the original feature map of the defect image to obtain multiple target-granularity feature maps, including:

[0109] Step 302: Determine the particle size division scheme for each particle size; the particle size division scheme includes the particle size division dimensions.

[0110] For example, in feature map processing, it is necessary to determine the granularity of the partitioning. Different partitioning sizes can extract feature information at different levels from the feature map. The partitioning size should be determined in conjunction with the specific task objective. When we need to obtain the overall features or macroscopic structure of the feature map, a larger partitioning size, i.e., coarse-grained partitioning, is used. For example, in object detection tasks, if we want to quickly locate the approximate position of a large target, coarse-grained partitioning can efficiently reduce the computational load and quickly grasp the overall situation of the feature map. If the focus is on capturing the detailed information of the feature map, such as identifying small targets or texture features in an image, a smaller partitioning size, i.e., fine-grained partitioning, is needed to further explore the local details of the feature map and provide a richer feature description.

[0111] In this embodiment, a multi-granularity generator consisting of multiple sub-generators is used to divide the original feature map into blocks. That is, the original feature map is input into the multi-granularity generator, and multiple sub-generators are used to divide the original feature map.

[0112] Step 304: Divide the original feature map into multiple granularities according to each granularity division scheme to obtain the initial feature correspondence; the initial feature correspondence includes the correspondence between different granularity feature blocks and the initial spatial coordinates.

[0113] For example, when performing granular segmentation on the original feature map, the original feature map is divided into granular feature blocks of different granularities according to different granularity segmentation schemes. If coarse-grained segmentation is used, the original feature map is divided into fewer but larger granular feature blocks; while if coarse-grained segmentation is used, more but smaller granular feature blocks are obtained.

[0114] Furthermore, during the partitioning process, it is necessary to record the positions of feature blocks of different granularities in the corresponding original feature maps, i.e., the initial spatial coordinates. Therefore, each granular feature block is associated with a specific spatial coordinate, thus forming the initial feature correspondence.

[0115] Step 306: Perform sequence processing on the initial feature correspondence to obtain the target feature correspondence; the target feature correspondence includes the correspondence between different granularity feature blocks and the reconstructed spatial coordinates.

[0116] The information contained in the initial feature correspondence may have certain limitations. Therefore, in order to further optimize the feature information, the initial feature correspondence can be further processed to provide strong support for subsequent more accurate defect analysis and identification.

[0117] For example, to make the original feature map after granular segmentation contain richer local information, it is necessary to randomly shuffle the granular feature blocks, that is, to disrupt the spatial distribution of the granular feature blocks. This can be achieved by randomly shuffling the arrangement order of the granular feature blocks, i.e., randomly shuffling the correspondence between the new granular feature blocks and their spatial coordinates, to obtain recombined spatial coordinates. These recombined spatial coordinates then form target feature correspondences with different granular feature blocks, thereby making the spatial distribution of feature blocks more dispersed and diverse, and enriching the local information.

[0118] In this embodiment, based on the determined granularity division scheme, the original feature map is divided into multiple granularities to obtain initial feature correspondences. This allows for precise location of feature blocks of different granularities in the original space, thereby helping to understand the distribution of defect features. Then, the initial feature correspondences are processed sequentially to obtain target feature correspondences, achieving spatial coordinate reorganization. This makes the feature information more orderly, facilitating improvements in the accuracy and efficiency of defect detection and analysis.

[0119] Since each target granularity feature map has the same size, it may not be possible to fully extract all the information from the defect features. Therefore, multiple target granularity feature maps can be further processed to achieve more effective feature fusion.

[0120] In one embodiment, such as Figure 4 As shown, the preset granularity fusion scheme includes a preset exchange size. Based on the preset granularity fusion scheme, cross-granularity fusion is performed on all target granularity feature maps to obtain a fused granularity feature set, including:

[0121] Step 402: For each feature map, based on the preset exchange size, select exchange feature blocks from the two target granularity feature maps in the feature map; the size of the exchange feature block matches the preset exchange size.

[0122] For example, a preset exchange size is determined before cross-granularity fusion, thus providing a fusion standard for cross-granularity fusion. Therefore, when processing feature maps, exchange feature blocks need to be selected from the two target granularity feature maps respectively, and the size of the selected exchange feature blocks must match the preset exchange size. For example, if the preset exchange size is set to 8×8, granular feature blocks with a size of 8×8 are selected from the two target granularity feature maps respectively, which are then the exchange feature blocks.

[0123] Step 404: Replace the exchanged feature blocks of the two target granularity feature maps in the feature map group to obtain the fused feature map group corresponding to the feature map group; wherein, the fused granularity feature set includes the fused feature map group of each feature map group.

[0124] For example, the exchanged feature blocks selected from two target granularity feature maps are permuted. Specifically, the exchanged feature blocks are extracted from the two target granularity feature maps to obtain target granularity feature maps with blank positions. The exchanged feature blocks from one target granularity feature map are then used to fill the blank positions in the other target granularity feature map. Similarly, the exchanged feature blocks from the other target granularity feature map are used to fill the blank positions in the first target granularity feature map. This results in a fused feature map group corresponding to the feature map group. The fused feature map group can more comprehensively reflect the defect features.

[0125] In this embodiment, by selecting exchange feature blocks from the target granularity feature map according to a preset exchange size, appropriate granularity feature blocks can be accurately selected for operation. Then, the exchange feature blocks of the two target granularity feature maps are permuted to obtain a fused feature map group, which enables information fusion between different feature maps, enriches feature representation, uncovers potential connections between features, and enhances feature diversity and complementarity.

[0126] To further improve the accuracy and controllability of feature exchange, and to make the fused features more targeted and effective, it is possible to determine the starting point of the regions of the two target granularity feature maps, combine this with a preset exchange size to determine the exchange region, and then select specific first and second granularity feature blocks as exchange feature blocks.

[0127] In one embodiment, such as Figure 5 As shown, based on a preset exchange size, exchanged feature blocks are selected from two target-granularity feature maps in the feature map group, including:

[0128] Step 502: Determine the starting point of the first region of the first granularity feature map and the starting point of the second region of the second granularity feature map; the first granularity feature map is a target granularity feature map of the feature map group, and the second granularity feature map is another target granularity feature map of the feature map group.

[0129] For example, the feature map group package specifically has two target granularity feature maps with different granularities, referred to in this embodiment as the first granularity feature map and the second granularity feature map, respectively.

[0130] To filter the exchange feature blocks, it is necessary to select the starting points of the permutation regions that match the preset exchange size on the first granularity feature map and the second granularity feature map, respectively. These are the first region starting point and the second region starting point. The first and second region starting points are randomly selected, and they do not have to be the same.

[0131] Step 504: Determine the first exchange region based on the preset exchange size and the starting point of the first region, and select the first granularity feature block located within the target exchange region from the first granularity feature map.

[0132] For example, starting from the first region, the first exchange region is divided on the first granular feature map according to the region range specified by the preset exchange size. Then, the granular feature blocks in the first exchange region of the first granular feature map are used as the granular feature blocks to be replaced, i.e., the first granular feature blocks. The selection of the replacement region adopts a uniformly distributed random non-repeating sampling strategy to avoid the replacement being concentrated in a specific or central region, so that all regions of the feature map have an equal chance to be selected.

[0133] Step 506: Determine the second exchange region based on the preset exchange size and the starting point of the second region, and filter the second granular feature blocks located within the second exchange region from the second granular feature map; wherein, the exchange feature blocks include the first granular feature blocks and the second granular feature blocks of the feature map group.

[0134] For example, starting from the second region, the exchange region, i.e., the second exchange region, is divided on the second granularity feature map according to the region range specified by the preset exchange size. Then, the granular feature blocks in the second exchange region of the second granularity feature map are used as the granular feature blocks to be replaced, i.e., the second granular feature blocks. The second exchange region and the first exchange region can be different regions.

[0135] Then, the first granularity feature block and the second granularity feature block are used as exchange feature blocks for this feature map group.

[0136] In this embodiment, the starting points of the regions in the first and second granularity feature maps are clearly defined, laying the foundation for accurate selection of exchanged feature blocks and enabling precise positioning of the starting position of feature exchange. Determining the exchange region based on the preset exchange size and region starting point ensures that the selected feature blocks meet spatial size requirements, enhancing the standardization and controllability of feature exchange. Selecting the first and second granularity feature blocks from the two granularity feature maps as exchanged feature blocks enables effective replacement of specific region features between different feature maps. This helps overcome the limitations of a single feature map, promotes the fusion and complementarity of feature information, and enhances the richness and diversity of feature expression.

[0137] In one embodiment, such as Figure 6 As shown, the exchanged feature blocks of the two target granularity feature maps in the feature map group are permuted to obtain the fused feature map group corresponding to the feature map group, including:

[0138] Step 602: Generate a first mask for a first granularity feature map based on the first exchange region, and generate a second mask for a second granularity feature map based on the second exchange region.

[0139] For example, after determining the first and second exchange regions, masks corresponding to the two exchange regions can be generated. For the first granularity feature map, based on the location and range of the first exchange region, the region is marked on the map to form the first mask. The mask is usually represented by binary values, with the exchange region being 0 (i.e., excluding the original value) and the others being 1 (i.e., retaining the original value). In one embodiment, the first mask is specifically:

[0140]

[0141] in, These represent the first granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, This is the first exchange area.

[0142] Similarly, a second mask is generated for the second granularity feature map based on the second exchange region. In one embodiment, the second mask is specifically:

[0143]

[0144] in, These represent the second granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, This is the second exchange area.

[0145] Step 604: Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, perform a weighted operation on the first granularity feature map to obtain the first fused feature map.

[0146] For example, the first mask defines the regions participating in the exchange in the first granularity feature map, while the second mask defines the regions participating in the exchange in the second granularity feature map. During weighted calculations, the first and second masks serve as the basis for weighting. For regions in the first granularity feature map that correspond to a value of 0 in the first mask, a weighted combination is performed by combining the feature information of the regions corresponding to the second mask in the second granularity feature map. Regions with a mask value of 1 retain the original features of the first granularity feature map.

[0147] By using weighted operations, effective feature fusion can be achieved, ultimately yielding a first fused feature map, thus enhancing feature representation capabilities. In one embodiment, the specific calculation method for the first fused feature map is as follows:

[0148]

[0149] in, This is the first fused feature map. This is the first granularity feature map. This is the second granularity feature map. As the first mask, This is the second mask.

[0150] Step 606: Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, perform a weighted operation on the second granularity feature map to obtain the second fused feature map; wherein, the fused feature map group includes the first fused feature map and the second fused feature map.

[0151] For example, based on the processing of the first granularity feature map, similarly, for the second granularity feature map, a weighted operation is performed on the corresponding part of the first granularity feature map in the region where the value is 1, using the second mask as a basis, to generate the second fused feature map.

[0152] Then the first fused feature map and the second fused feature map together constitute a fused feature map group.

[0153] In some embodiments, the first granularity feature map is represented as The second granularity feature map is represented as And both the first and second switching areas are The size of the region is specifically denoted as:

[0154]

[0155]

[0156] in, This is the first exchange area. For the second exchange area, The x-axis of the first exchange region is the first granularity feature map. Position coordinates in the horizontal direction The vertical coordinate of the first exchange region is the first granularity feature map. Position coordinates in the vertical direction The x-axis of the second exchange region is the second granularity feature map. Position coordinates in the horizontal direction The vertical coordinate of the second exchange region is the second granularity feature map. Position coordinates in the vertical direction.

[0157] The first and second fused feature maps obtained by cross-granularity fusion are as follows:

[0158]

[0159]

[0160] in, This is the first fused feature map. This is the second fused feature map. As the first mask, This is the second mask.

[0161] Masks are typically represented using binary values, with swapped regions set to 0 and all others to 1. The specific representations of the first and second masks are shown below:

[0162]

[0163]

[0164] In this embodiment, by generating a mask corresponding to the exchange region in the granularity feature map, the region participating in feature fusion can be accurately defined, making feature exchange more targeted. Then, by using weighted operations, information interaction and fusion between the two feature maps are realized, effectively integrating the advantages of feature maps of different granularities and enriching feature representation.

[0165] To more clearly illustrate the process of constructing feature prototypes, it is necessary to further refine the operation, clarify that the fusion granularity feature set is composed of the fusion feature map group of each feature map group, and explain in detail the specific steps of constructing feature prototype vectors based on the fusion feature map group.

[0166] In one embodiment, such as Figure 7 As shown, the fusion granularity feature set includes the fusion feature map group of each feature map group; based on the fusion granularity feature set corresponding to the defect image of the same defect type, feature prototypes are constructed for the same defect type to obtain the feature prototype vector corresponding to the same defect type, including:

[0167] Step 702: For each defect type, determine the fusion feature map in all fusion feature map groups corresponding to the defect type.

[0168] For example, for each specific defect type, all defect images of that defect type are collected and identified, and then filtered and organized according to the defect type label, grouping images with the same defect features into one category. Then, for each defect image in that defect type, the corresponding fusion feature map needs to be selected from all fusion feature map groups.

[0169] Step 704: Based on the ridge regression model, perform feature reconstruction on the fused feature maps in all fused feature map groups of the same defect to obtain the feature prototypes corresponding to the same defect type.

[0170] For example, all fused feature maps of the same defect are aligned spatially and stacked to form a three-dimensional feature tensor with dimensions of sample number × feature map height × feature map width. Each spatial location in this tensor corresponds to a sequence of feature values ​​from multiple fused feature maps. Then, a ridge regression optimization problem is constructed. For each spatial location (i.e., each pixel) in the feature tensor, a vector is formed from the feature values ​​of all fused feature maps at that location as the observation value, and the average feature value at that location across all samples is used as the target to be reconstructed. By minimizing the ridge regression loss function, which includes a squared error term and an L2 regularization term, a weight vector capable of stably reconstructing the target feature value from the observed feature vector is fitted. Subsequently, the trained ridge regression model is used to reconstruct the feature vector at each spatial location, generating a smooth and representative feature map, which is then used as the feature prototype for this defect type.

[0171] In this embodiment, determining the corresponding fusion feature map for each defect type allows for precise focus on the feature information related to each defect type, avoiding interference from irrelevant information. Furthermore, aggregating the fusion feature maps integrates the feature information scattered across different fusion feature maps, forming a feature prototype that represents the defect type, thus improving the accuracy and efficiency of defect identification.

[0172] Based on the construction of feature prototypes for each defect type, the target defect type of the candidate detection feature map can be determined using the feature prototypes. The following are detailed steps for determining the target defect type.

[0173] In one embodiment, such as Figure 8 As shown, based on the feature prototypes corresponding to each defect type in multiple defect images, the target defect type of the candidate detection feature map is determined, including:

[0174] Step 802: Transform the candidate detection feature map to obtain candidate detection features.

[0175] For example, to facilitate determining the defect type corresponding to a candidate detection feature map, it is necessary to transform the candidate feature map into a candidate detection feature that resides in the same feature space as the feature prototype. Feature extraction algorithms, such as principal component analysis and local binary pattern recognition, can be used to extract key features from the candidate detection feature map. Then, these extracted key features are arranged and combined according to certain rules to form the candidate detection features.

[0176] Step 804: For each defect type, calculate the similarity between the candidate detection features and the feature prototype to obtain the feature similarity.

[0177] For example, to determine which type of defect might exist in the image to be detected corresponding to a candidate detection feature, it is necessary to calculate the similarity between the candidate detection feature and each feature prototype. Similarity calculation can employ methods such as cosine similarity or Euclidean distance. Taking cosine similarity as an example, it measures the similarity between two features. The calculated feature similarity value can intuitively reflect the degree of closeness between the candidate detection feature and the typical features of each defect type.

[0178] Step 806: Determine the target defect type as the defect type corresponding to the largest feature similarity.

[0179] For example, each defect type corresponds to a feature similarity value for the image to be detected, reflecting the degree to which potential defects in the image are close to the typical features of each defect type. By comparing all feature similarities, the largest one is identified. The largest feature similarity means that the candidate detection feature best matches the typical features of the defect type corresponding to that similarity. Therefore, the defect type corresponding to this largest feature similarity is determined as the target defect type.

[0180] In this embodiment, the candidate detection feature map is transformed to obtain features. Then, the similarity between the candidate detection features and the feature prototype of each defect type is calculated. By comparing the similarity, it is possible to quickly and accurately determine which defect type the candidate sample is most similar to, thereby improving the accuracy and reliability of detection and reducing the false positive rate.

[0181] In one exemplary embodiment, a defect detection method for integrated circuit images is provided, the method comprising the following steps:

[0182] Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected.

[0183] For each defect image, determine the granularity division scheme for each granularity; the granularity division scheme includes the granularity division size.

[0184] The defect features are divided into granularities according to each granularity division scheme to obtain the initial feature correspondence; the initial feature correspondence includes the correspondence between different granularity feature blocks and the initial spatial coordinates.

[0185] The initial feature correspondence is processed sequentially to obtain the target feature correspondence; the target feature correspondence includes the correspondence between feature blocks of different granularities and the reconstructed spatial coordinates.

[0186] For each feature map group, determine the starting point of the first region of the first granularity feature map and the starting point of the second granularity feature map; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps, the first granularity feature map is one target granularity feature map in the feature map group, and the second granularity feature map is another target granularity feature map in the feature map group.

[0187] The first exchange region is determined based on the preset exchange size and the starting point of the first region, and the first granularity feature block located within the target exchange region is selected from the first granularity feature map.

[0188] The second exchange region is determined based on the preset exchange size and the starting point of the second region, and the second granularity feature blocks located within the second exchange region are selected from the second granularity feature map.

[0189] The exchange feature block includes a first-granularity feature block and a second-granularity feature block of the feature map group; the size of the exchange feature block matches the preset exchange size.

[0190] A first mask for generating a first granularity feature map based on a first exchange region, and a second mask for generating a second granularity feature map based on a second exchange region.

[0191] Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the first granularity feature map to obtain the first fused feature map.

[0192] Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the second granularity feature map to obtain the second fused feature map.

[0193] The fused feature map set includes a first fused feature map and a second fused feature map, and the fused granularity feature set includes fused feature map sets of each feature map set, with the first fused feature map... Second fusion feature map The specific calculation method is as follows:

[0194]

[0195]

[0196]

[0197]

[0198] In the above calculation method, These represent the first granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, These represent the second granularity feature maps respectively. Position coordinates in the horizontal and vertical directions This is the first exchange area. This is the second exchange area.

[0199] For each defect type, determine the fused feature map in all fused feature map groups corresponding to the defect type.

[0200] Based on the ridge regression model, feature reconstruction is performed on the fused feature maps in all fused feature map groups to obtain the feature prototypes corresponding to the defect types.

[0201] The candidate detection feature map is transformed to obtain candidate detection features.

[0202] For each defect type, the similarity between the candidate detection features and the feature prototype is calculated to obtain the feature similarity.

[0203] The target defect type is determined as the defect type corresponding to the highest feature similarity.

[0204] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0205] Based on the same inventive concept, this application also provides an integrated circuit image defect detection device for implementing the aforementioned integrated circuit image defect detection method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more integrated circuit image defect detection device embodiments provided below can be found in the limitations of the integrated circuit image defect detection method described above, and will not be repeated here.

[0206] In one exemplary embodiment, such as Figure 9 As shown, a defect detection device for integrated circuit images is provided, comprising: a data acquisition module 902, a granularity division module 904, a cross-granularity fusion module 906, a feature prototype construction module 908, and a defect determination module 910, wherein:

[0207] The data acquisition module 902 is used to acquire the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected.

[0208] The granularity segmentation module 904 is used to extract the granularity of the original feature map of each defect image to obtain multiple target granularity feature maps. The target granularity feature maps include the correspondence between different spatial coordinates and different granularity feature blocks.

[0209] The cross-granularity fusion module 906 is used to perform cross-granularity fusion on the feature map group according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from multiple target granularity feature maps.

[0210] The feature prototype construction module 908 is used to perform feature reconstruction on the same defect type based on the fusion granularity feature set corresponding to the defect image of the same defect type, and obtain the feature prototype corresponding to the same defect type.

[0211] The defect determination module 910 is used to determine the target defect type of the candidate detection feature map based on the feature prototypes corresponding to each defect type in multiple defect images.

[0212] In one embodiment, the data acquisition unit is used to determine the granularity division scheme for each granularity; the granularity division scheme includes the granularity division size; the defect features are granularized according to each granularity division scheme to obtain an initial feature correspondence; the initial feature correspondence includes the correspondence between different granularity feature blocks and initial spatial coordinates; the initial feature correspondence is processed sequentially to obtain a target feature correspondence; the target feature correspondence includes the correspondence between different granularity feature blocks and reconstructed spatial coordinates.

[0213] In one embodiment, the granularity partitioning unit is used to select exchange feature blocks from two target granularity feature maps in each feature map group based on a preset exchange size; the exchange feature blocks include granularity feature blocks whose sum of spatial dimensions matches the preset exchange size; the exchange feature blocks of the two target granularity feature maps in the feature map group are permuted to obtain a fused feature map group corresponding to the feature map group; wherein, the fused granularity feature set includes the fused feature map group of each feature map group.

[0214] In one embodiment, the granularity division unit is further configured to determine a first region starting point of a first granularity feature map and a second region starting point of a second granularity feature map; the first granularity feature map is a target granularity feature map of a feature map group, and the second granularity feature map is another target granularity feature map of the feature map group; a first exchange region is determined based on a preset exchange size and the first region starting point, and a first granularity feature block located within the target exchange region is selected from the first granularity feature map; a second exchange region is determined based on the preset exchange size and the second region starting point, and a second granularity feature block located within the second exchange region is selected from the second granularity feature map; wherein, the exchange feature block includes the first granularity feature block and the second granularity feature block of the feature map group.

[0215] In one embodiment, the cross-granularity fusion unit is configured to generate a first mask for a first granularity feature map based on a first exchange region, and a second mask for a second granularity feature map based on a second exchange region; perform a weighted operation on the first granularity feature map according to the first granularity feature map, the second granularity feature map, the first mask, and the second mask to obtain a first fused feature map; and perform a weighted operation on the second granularity feature map according to the first granularity feature map, the second granularity feature map, the first mask, and the second mask to obtain a second fused feature map; wherein the fused feature map group includes the first fused feature map. Second fusion feature map The specific calculation method is as follows:

[0216]

[0217]

[0218]

[0219]

[0220] In the above calculation method, These represent the first granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, These represent the second granularity feature maps respectively. Position coordinates in the horizontal and vertical directions This is the first exchange area. This is the second exchange area.

[0221] In one embodiment, the feature prototype construction unit is used to determine the fused feature map in all fused feature map groups corresponding to each defect type for each defect type; and to perform feature reconstruction on the fused feature map in all fused feature map groups of the same defect according to the ridge regression model to obtain the feature prototype corresponding to the defect type.

[0222] In one embodiment, the defect determination unit is used to transform the candidate detection feature map to obtain candidate detection features; for each defect type, the similarity between the candidate detection features and the feature prototype is calculated to obtain feature similarity; and the target defect type is determined as the defect type corresponding to the largest feature similarity.

[0223] Each module in the aforementioned integrated circuit image defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0224] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores the original feature maps of defect images and candidate detection feature map data of candidate regions in the image to be inspected. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a defect detection method for integrated circuit images.

[0225] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0226] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0227] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0228] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0229] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0230] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0231] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0232] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for defect detection in integrated circuit images, characterized in that, The method includes: Obtain the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected; For each of the aforementioned defect images, multi-granularity extraction is performed on the original feature map of the defect image to obtain multiple target granularity feature maps; the target granularity feature maps include feature blocks with different spatial coordinates and different granularity sizes; The feature map group is fused across granularities according to a preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from the plurality of target granularity feature maps; Based on the fusion granularity feature set corresponding to the defect image of the same defect type, feature reconstruction is performed on the same defect type to obtain the feature prototype corresponding to the same defect type; Based on the feature prototypes corresponding to each defect type in the multiple defect images, the target defect type of the candidate detection feature map is determined.

2. The method according to claim 1, characterized in that, The step of performing multi-granularity extraction on the original feature map of the defect image to obtain multiple target granularity feature maps includes: Determine the particle size division scheme for each particle size; the particle size division scheme includes the particle size division dimensions; The original feature map is divided into multiple granularities according to the granularity division schemes described above to obtain the initial feature correspondence; the initial feature correspondence includes the correspondence between different granularity feature blocks and the initial spatial coordinates. The initial feature correspondence is processed sequentially to obtain the target feature correspondence; the target feature correspondence includes the correspondence between different granularity feature blocks and the recombined spatial coordinates.

3. The method according to claim 1, characterized in that, The preset granularity fusion scheme includes a preset exchange size. The step of performing cross-granularity fusion on all the target granularity feature maps according to the preset granularity fusion scheme to obtain a fused granularity feature set includes: For each of the aforementioned feature map groups, based on the preset exchange size, exchange feature blocks are selected from the two target granularity feature maps in the feature map group; the size of the exchange feature block matches the preset exchange size. The exchanged feature blocks of the two target granularity feature maps in the feature map group are replaced to obtain the fused feature map group corresponding to the feature map group; The fusion granularity feature set includes fusion feature map sets of each of the feature map sets.

4. The method according to claim 3, characterized in that, The step of selecting exchange feature blocks from two target granularity feature maps in the feature map group based on the preset exchange size includes: Determine the starting point of the first region of the first granularity feature map and the starting point of the second region of the second granularity feature map; the first granularity feature map is one of the target granularity feature maps in the feature map group, and the second granularity feature map is another target granularity feature map in the feature map group; A first exchange region is determined based on the preset exchange size and the starting point of the first region, and a first granular feature block located within the target exchange region is selected from the first granular feature map; The second exchange region is determined based on the preset exchange size and the starting point of the second region, and the second granular feature block located within the second exchange region is selected from the second granular feature map; The exchanged feature block includes the first granularity feature block and the second granularity feature block of the feature map group.

5. The method according to claim 4, characterized in that, The step of permuting the exchanged feature blocks of the two target granularity feature maps in the feature map group to obtain the fused feature map group corresponding to the feature map group includes: A first mask for generating the first granularity feature map based on the first exchange region, and a second mask for generating the second granularity feature map based on the second exchange region; Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the first granularity feature map to obtain the first fused feature map; Based on the first granularity feature map, the second granularity feature map, the first mask, and the second mask, a weighted operation is performed on the second granularity feature map to obtain the second fused feature map; The fused feature map includes a first fused feature map. Second fusion feature map The specific calculation method is as follows: In the above calculation method, These represent the first granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, These represent the second granularity feature maps respectively. Position coordinates in the horizontal and vertical directions, This is the first exchange area. This is the second exchange area.

6. The method according to claim 1, characterized in that, The fusion granularity feature set includes fusion feature map sets of each of the feature map groups; the step of performing feature reconstruction on the same defect type based on the fusion granularity feature set corresponding to the defect image of the same defect type to obtain the feature prototype corresponding to the same defect type includes: For each defect type, determine the fused feature map in all fused feature map groups corresponding to the defect type; Based on the ridge regression model, feature reconstruction is performed on the fused feature maps in all fused feature map groups of the same defect to obtain the feature prototype corresponding to the same defect type.

7. The method according to claim 1, characterized in that, The step of determining the target defect type of the candidate detection feature map based on the feature prototypes corresponding to each defect type of the multiple defect images includes: The candidate detection feature map is transformed to obtain candidate detection features; For each of the aforementioned defect types, the similarity between the candidate detection features and the feature prototype is calculated to obtain the feature similarity. The target defect type is determined to be the defect type corresponding to the highest feature similarity.

8. A defect detection device for integrated circuit images, characterized in that, The device includes: The data acquisition module is used to acquire the original feature maps of multiple defect images and the candidate detection feature maps of candidate regions in the image to be detected; The multi-granularity partitioning module is used to extract multiple target granularity feature maps from the original feature maps of each defect image. The target granularity feature maps include the correspondence between different spatial coordinates and different granularity feature blocks. The cross-granularity fusion module is used to perform cross-granularity fusion on the feature map group according to the preset granularity fusion scheme to obtain a fused granularity feature set; the feature map group includes any two target granularity feature maps from the plurality of target granularity feature maps. The feature prototype construction module is used to construct feature prototypes for the same defect type based on the fusion granularity feature set corresponding to the defect image of the same defect type, so as to obtain the feature prototypes corresponding to the same defect type. The defect determination module is used to determine the target defect type of the candidate detection feature map based on the feature prototypes corresponding to each defect type of the multiple defect images.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images