Method, system, electronic device, storage medium and program product for image recognition

Abstract

The present disclosure provides a method, system, electronic device, storage medium and program product for image recognition. The method comprises: segmenting a target image to generate a mask set, wherein each mask in the mask set is a candidate mask, and each candidate mask is used to identify a corresponding candidate region in the target image; and using an image information based matching method to select a matching mask that matches a reference mask from the candidate masks, so as to obtain a target region in the target image that matches a reference region in a reference image identified by the reference mask.

Description

Technical Field

[0001] This disclosure generally relates to integrated circuits, and more specifically to methods, systems, electronic devices, computer-readable storage media, and computer program products for image recognition. Background Technology

[0002] In the semiconductor manufacturing process, processes such as Automated Optical Inspection (AOI), lithography machine alignment, die-to-database (D2DB) comparison, and package appearance inspection all rely heavily on high-precision image recognition and matching technologies.

[0003] In scenarios such as semiconductor images, industrial inspection images, and microscopic images, the target image structure often has problems such as complex shape, local missing parts, blurred boundaries, obvious texture changes, and a large number of candidate regions. Traditional image recognition and matching methods have drawbacks such as insufficient matching accuracy, high false detection rate, and large computational cost. Summary of the Invention

[0004] According to exemplary embodiments of this disclosure, a scheme for image recognition is provided to at least partially overcome the above or other potential drawbacks.

[0005] In a first aspect of this disclosure, a method for image recognition is provided. The method includes: segmenting a target image to generate a mask set, wherein each mask in the mask set serves as a candidate mask, and the candidate masks are used to identify corresponding candidate regions in the target image; and using an image-information-based matching method to select a matching mask from the candidate masks that matches a reference mask, thereby obtaining a target region in the target image that matches a reference region in the reference image identified by the reference mask.

[0006] In some embodiments, using an image information-based matching method to select a matching mask that matches a reference mask from candidate masks to obtain a target region in the target image that matches a reference region identified by the reference mask in the reference image includes: determining the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region; and determining the candidate mask with the highest similarity among the candidate masks as the matching mask.

[0007] In some embodiments, determining the similarity between a candidate mask and a reference mask includes determining a weighted sum of at least two of the following: image embedding similarity between the candidate mask and the reference mask, grayscale distribution similarity between the candidate mask and the reference mask, and boundary gradient direction similarity between the candidate mask and the reference mask.

[0008] In some embodiments, the method further includes: preserving the image embedding features of the target image when segmenting the target image; performing region pooling on the image embedding features of the target image based on a candidate mask to generate a local feature vector corresponding to the candidate mask; and using the local feature vector and the image embedding features of the reference region to calculate the image embedding similarity.

[0009] In some embodiments, structural information includes at least one of the following: structural type, area, center location, number of connected domains, and aspect ratio.

[0010] In some embodiments, determining the similarity between a candidate mask and a reference mask based on a comparison of the structural information of a reference region and the structural information of a candidate region includes: calculating the similarity between the candidate mask and the reference mask based on the fact that the structure of the reference region is a solid structure and the structure of the candidate region is a solid structure; calculating the similarity between the candidate mask and the reference mask based on the fact that the structure of the reference region is a ring structure and the structure of the candidate region is a ring structure; and performing ring candidate reconstruction on the candidate mask to generate a reconstructed candidate mask based on the fact that the structure of the reference region is a ring structure and the structure of the candidate region is a ring candidate structure, and calculating the similarity between the reconstructed candidate mask and the reference mask.

[0011] In some embodiments, performing annular candidate reconstruction on the candidate mask to generate a reconstructed candidate mask, and calculating the similarity between the reconstructed candidate mask and the reference mask includes: selecting a first candidate mask and a second candidate mask from the candidate masks corresponding to the candidate regions determined to be annular candidate structures, wherein the area of ​​the first candidate mask is smaller than that of the second candidate mask and the pattern of the first candidate mask is contained within the pattern of the second candidate mask; and generating a reconstructed candidate mask as a new candidate mask by subtracting the first candidate mask from the second candidate mask, and calculating the similarity between the new candidate mask and the reference mask.

[0012] In some embodiments, determining the similarity between a candidate mask and a reference mask based on a comparison of the structural information of a reference region and the structural information of a candidate region includes: extracting multiple contours from a mask to be processed, the mask to be processed including a candidate mask and a reference mask; and when there is a contour hierarchy relationship among the multiple contours, determining that the structure of the region identified by the mask to be processed is a ring structure; and when there is no contour hierarchy relationship among the multiple contours, selecting the contour with the largest area among the multiple contours as the largest contour.

[0013] In some embodiments, the method further includes: calculating the roundness of the maximum contour; and when the roundness of the maximum contour is lower than a preset threshold, determining that the structure of the region identified by the mask to be processed is a ring candidate structure; and when the roundness of the maximum contour is not lower than the preset threshold, determining that the structure of the region identified by the mask to be processed is a solid structure.

[0014] In some embodiments, determining the candidate mask with the highest similarity among the candidate masks as the matching mask includes: calculating a decision function value using similarity and area similarity; and determining the candidate mask with the highest decision function value as the matching mask.

[0015] In some embodiments, the decision function value is calculated using the following formula:

[0016]

[0017] in, Let k be the decision function of the candidate mask. This represents the similarity between the candidate mask k and the reference mask. This represents the area of ​​candidate mask k. Indicates the area of ​​the reference mask. Weighting coefficients representing similarity and The weighting coefficients representing area similarity.

[0018] In some embodiments, the matching mask is applied to at least one of the following: defect detection, target localization, image alignment, and structural analysis.

[0019] In a second aspect of this disclosure, a system for image recognition is provided. The system includes: a candidate segmentation generation module for segmenting a target image to generate a mask set, wherein each mask in the mask set serves as a candidate mask, and the candidate masks are used to identify corresponding candidate regions in the target image; and a structure matching decision module for selecting a matching mask from the candidate masks that matches a reference mask, to obtain a target region in the target image that matches a reference region in the reference image identified by the reference mask.

[0020] In some embodiments, the system further includes an image input module for acquiring a target image, a reference image, and a reference mask.

[0021] In some embodiments, the system further includes an embedding retention module for retaining the image embedding features of the target image.

[0022] In some embodiments, the system further includes a reference structure template extraction module, used to extract structural information of a reference region identified by a reference mask in a reference image.

[0023] In some embodiments, the system further includes a similarity calculation module, used to determine the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region.

[0024] In some embodiments, the system further includes: a result output module for outputting a matching mask, the matching mask being used for at least one of the following: defect detection, target localization, image alignment, and structural analysis.

[0025] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method according to a first aspect of this disclosure.

[0026] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0027] In a fifth aspect of this disclosure, a computer program product is provided, which includes program code that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0028] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0030] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0031] Figure 2 A flowchart of a method for image recognition according to some embodiments of the present disclosure is shown;

[0032] Figure 3 A schematic diagram of a system for image recognition according to an embodiment of the present disclosure is shown;

[0033] Figure 4 A flowchart of a method for image recognition according to some embodiments of the present disclosure is shown;

[0034] Figure 5 A flowchart of a method for determining the structure type according to an embodiment of the present disclosure is shown;

[0035] Figure 6 An example schematic diagram of similarity calculation according to an embodiment of the present disclosure is shown;

[0036] Figure 7 A block diagram of an electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0037] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0038] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0039] As briefly mentioned earlier, in the semiconductor manufacturing process, automated optical inspection, lithography machine alignment, chip-to-database comparison, and package appearance inspection all heavily rely on high-precision image recognition and matching technologies. Traditional image recognition and matching methods suffer from insufficient matching accuracy, high false detection rates, and high computational costs. Especially for target images with ring structures or complex topological relationships, the masks generated by existing automatic segmentation methods often have gaps or incompleteness, greatly increasing the difficulty of target matching.

[0040] However, traditional methods still have certain limitations in practical applications. Traditional methods typically rely on template matching, edge comparison, or single-feature similarity calculation, focusing only on local structural information. They struggle to simultaneously integrate the overall semantic information of the target, local grayscale distribution information, and boundary direction features. When the target image exhibits scale changes, deformation, defects, noise interference, or texture variations, this can easily lead to low matching accuracy and high false positive rates. Furthermore, when processing images to be matched, traditional methods usually require calculations on a large number of candidate regions within the image. The lack of a mechanism for effectively filtering candidate regions based on the structural features of the reference target results in many regions unrelated to the target participating in the subsequent matching process, increasing computational resource consumption and reducing the stability and processing efficiency of the matching process.

[0041] Furthermore, as mentioned earlier, traditional image recognition and matching methods suffer from insufficient matching accuracy, high false detection rates, and high computational costs. For target images with ring structures, hollow structures, or complex topological relationships, the masks generated by existing automatic segmentation methods are often fragmented or incomplete, greatly increasing the difficulty of target matching and making it difficult to accurately obtain the complete target region. For example, when the target region has breaks, discontinuous boundaries, or changes in the representational relationship between the target region and the background region (e.g., changes in grayscale contrast, changes in foreground-background identification), the structural recognition of the target image fails, thus affecting the accuracy of subsequent matching results. In addition, traditional schemes often require repeated image encoding or feature extraction operations during candidate region generation, feature extraction, and similarity calculation, failing to fully utilize existing intermediate feature information. When the target image resolution is high or the number of candidate regions is large, there are drawbacks such as high computational cost, slow processing speed, and high cost, thus limiting the application of image recognition and matching methods in practical engineering scenarios.

[0042] In view of this, this disclosure provides an improved method for image recognition.

[0043] According to embodiments of this disclosure, a method for image recognition is provided. The method includes: segmenting a target image to generate a mask set, wherein each mask in the mask set serves as a candidate mask, and the candidate masks are respectively used to identify corresponding candidate regions in the target image; and using an image-information-based matching method to select a matching mask from the candidate masks that matches a reference mask, thereby obtaining a target region in the target image that matches a reference region in the reference image identified by the reference mask.

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] Figure 1 An example environment 100 in which embodiments of this disclosure can be implemented is shown. For example... Figure 1As shown, the electronic device 110 can acquire a target image 102, a reference image 104, and a reference mask 106. The target image 102 is used to perform candidate mask generation and search, as well as mask matching and recognition; the reference image 104 is used to provide structural information of the reference region; and the reference mask 106 is used to identify the reference region in the reference image 104. In some embodiments, the target image 102 is an RGB three-channel color image. In some embodiments, the reference mask 106 is a binary mask spatially aligned with the reference image 104. Spatial alignment means that the reference mask and the reference image have the same spatial dimensions, and the pixel positions in the reference mask correspond one-to-one with the pixel positions in the reference image. A first value (e.g., 1) in the binary mask is used to represent the reference region in the reference image, and a second value (e.g., 0) is used to represent the background region in the reference image. The electronic device 110 can output a matching mask 108. The matching mask 108 identifies the region in the target image 102 that corresponds to the reference region in the reference image 104.

[0046] Electronic device 110 can be any device with computing capabilities. As a non-limiting example, electronic device 110 can be any type of fixed electronic device, mobile electronic device, or portable electronic device, including but not limited to desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, multimedia computers, mobile phones, smart home devices, wearable electronic devices, etc. In some embodiments, all or some components of electronic device 110 can be distributed in the cloud. This disclosure does not limit the specific type of electronic device 110.

[0047] Figure 2 A flowchart of a method 200 for image recognition according to some embodiments of the present disclosure is shown. For ease of discussion, it will be combined with... Figure 1 Let's describe method 200. Figure 2 Method 200 can be found in Figure 1 The steps are executed at electronic device 110 and any suitable electronic device. Furthermore, the numbers in the flowchart do not indicate the order in which these steps are executed; some or all of these steps may be executed in parallel, or their execution order may be interchanged, and this disclosure does not limit this.

[0048] In box 202, the target image is segmented to generate a mask set, where each mask in the mask set serves as a candidate mask, and each candidate mask is used to identify a corresponding candidate region in the target image. In some embodiments, the target image can be segmented based on an image segmentation model to generate the mask set. In some embodiments, the image segmentation model may include a Segment Anything Model (SAM), a U-Net, a Mask Region-based Convolutional Neural Network (Mask R-CNN), and other models capable of generating mask instances. In some embodiments, when segmenting the target image, the image embedding features of the target image can be preserved; based on the candidate masks, the image embedding features of the target image can be subjected to region pooling to generate local feature vectors corresponding to the candidate masks. The local feature vectors preserve the image embedding features of the target image, thereby avoiding repeated feature encoding of the target image in subsequent processes, thus reducing computational overhead and improving overall efficiency.

[0049] In some embodiments, all masks in the mask set can be used as candidate masks. In some embodiments, preliminary screening of the masks in the mask set can also be performed: the structural information of the region identified by each mask in the mask set can be compared with the structural information of the reference region identified by the reference mask to generate a comparison result; and masks in the mask set whose comparison results meet predetermined conditions are determined as candidate masks. Preliminary screening can remove candidate regions that clearly do not conform to the structural characteristics of the reference region, thereby saving time and computational resources consumed in subsequent processing steps.

[0050] In some embodiments, structural information may include at least one of the following: structure type, region area, center location, number of connected components, and aspect ratio. The structure type distinguishes whether the structure is a solid structure, a ring structure, or other structures. The region area represents the number of pixels covered by the region or the actual spatial area. In some embodiments, the region area can be obtained by counting the number of pixels within the region. The center location represents the spatial location of the region in the image. In some embodiments, the geometric center of the region can be calculated based on the positions of the pixels within the region. The number of connected components represents the number of mutually independent sub-regions within the region. In some embodiments, the region can be analyzed for connectivity based on preset connectivity rules to identify multiple non-connected sub-regions, and the number of sub-regions can be counted as the number of connected components. In some embodiments, the connectivity rules may include four-neighbor connectivity or eight-neighbor connectivity. The aspect ratio represents the size ratio of the region in the horizontal and vertical directions. In some embodiments, the minimum bounding rectangle surrounding the target region can be determined, and the aspect ratio can be obtained based on the ratio of the width and height of the minimum bounding rectangle.

[0051] In some embodiments, the predetermined conditions include one or more of the following: the area of ​​the region is within a predetermined area range, the aspect ratio is within a predetermined aspect ratio range, the proximity between the candidate location and the target center location is within a predetermined proximity range, the consistency of the structure type, and the consistency of the number of connected components. In some embodiments, these predetermined conditions may be set manually or automatically determined by a computer according to preset rules, and this disclosure does not limit them.

[0052] In box 204, an image-based matching method is used to select a matching mask from candidate masks that matches a reference mask, thereby obtaining a target region in the target image that matches a reference region in the reference image identified by the reference mask. In some embodiments, selecting a matching mask from candidate masks that matches a reference mask to obtain a target region in the target image that matches a reference region in the reference image identified by the reference mask includes: determining the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region; and determining the candidate mask with the highest similarity among the candidate masks as the matching mask. In some embodiments, determining the similarity between the candidate mask and the reference mask may include: determining a weighted sum of at least two of the following: image embedding similarity between the candidate mask and the reference mask, grayscale distribution similarity between the candidate mask and the reference mask, and boundary gradient direction similarity between the candidate mask and the reference mask. In some embodiments, the image embedding similarity is calculated using the aforementioned local feature vectors and image embedding features. In some embodiments, image embedding similarity is used to represent the similarity between candidate regions and reference regions in deep feature space; grayscale distribution similarity is used to represent the consistency between candidate regions and reference regions in grayscale or texture distribution; and boundary gradient direction similarity is used to represent the consistency between candidate regions and reference regions in boundary morphology and local directional features.

[0053] In some embodiments, the structure type needs to be determined when comparing the structural information with the structural information of the candidate region identified by the candidate mask. In some embodiments, determining the structure type may include: extracting multiple contours from the mask to be processed, the mask including the candidate mask and the reference mask; and when a contour hierarchy relationship exists among the multiple contours, determining that the structure of the region identified by the mask to be processed is a ring structure; when no contour hierarchy relationship exists among the multiple contours, selecting the contour with the largest area among the multiple contours as the largest contour. A contour hierarchy relationship refers to the hierarchical relationship established between multiple contours based on geometric inclusion relationships. For a first contour and a second contour, if the closed region corresponding to the first contour contains the closed region corresponding to the second contour, then a contour hierarchy relationship exists between the first contour and the second contour, where the first contour is the parent contour of the second contour, and the second contour is the child contour of the first contour. Based on the contour hierarchy relationship between multiple contours, the nesting level of each contour in the mask can be determined, and further, whether the contour corresponds to a hole region can be determined based on the nesting level. In some embodiments, the roundness of the maximum contour is calculated; and when the roundness of the maximum contour is lower than a preset threshold, the structure of the region identified by the mask to be processed is determined to be a ring candidate structure; when the roundness of the maximum contour is not lower than the preset threshold, the structure of the region identified by the mask to be processed is determined to be a solid structure. A ring candidate structure indicates that the structure of the region identified by the mask is a broken or incomplete ring.

[0054] In some embodiments, the similarity between the candidate mask and the reference mask is calculated based on the fact that the structure of the reference region is a solid structure and the structure of the candidate region is a solid structure; the similarity between the candidate mask and the reference mask is calculated based on the fact that the structure of the reference region is a ring structure and the structure of the candidate region is a ring structure; and the similarity between the reconstructed candidate mask and the reference mask is calculated based on the fact that the structure of the reference region is a ring structure and the structure of the candidate region is a ring candidate structure.

[0055] In some embodiments, performing annular candidate reconstruction on the candidate mask to generate a reconstructed candidate mask, and calculating the similarity between the reconstructed candidate mask and the reference mask may include: selecting a first candidate mask and a second candidate mask from the candidate masks corresponding to the candidate regions determined to be annular candidate structures, wherein the area of ​​the first candidate mask is smaller than that of the second candidate mask and the pattern of the first candidate mask is contained within the pattern of the second candidate mask; and generating a reconstructed candidate mask as a new candidate mask by subtracting the first candidate mask from the second candidate mask, and calculating the similarity between the new candidate mask and the reference mask.

[0056] As previously mentioned, similarity includes image embedding similarity, grayscale distribution similarity, and boundary gradient direction similarity. In some embodiments, the various similarities can be combined to obtain a comprehensive similarity, and the candidate mask with the highest comprehensive similarity is selected as the candidate mask with the highest similarity. In some embodiments, combining the various similarities includes weighted summation of the various similarities. In some embodiments, only any two of the aforementioned similarities can be considered, and a weighted summation of these two similarities can be performed to obtain the comprehensive similarity. In some embodiments, similarity may also include other similarities, which are not limited in this disclosure.

[0057] In some embodiments, to reduce the risk of mismatches caused by factors such as the randomness of high matching scores of candidate masks or the instability of graphic boundaries, a final decision function can be constructed by further combining the proximity of the candidate region and the reference region in terms of area. That is, the decision function value is calculated using comprehensive similarity and area similarity; and the candidate mask with the largest decision function value is determined as the matching mask. In some embodiments, the decision function value is calculated using the following formula: (1) in, Let k be the decision function of the candidate mask. This represents the similarity between the candidate mask k and the reference mask. This represents the area of ​​candidate mask k. Indicates the area of ​​the reference mask. Weighting coefficients representing similarity and The weighting coefficients representing area similarity.

[0058] In some embodiments, the matching mask can be applied to at least one of the following: defect detection, target localization, image alignment, and structural analysis.

[0059] Advantageously, the image recognition method according to embodiments of this disclosure saves time and computational resources caused by repeated calculations by segmenting candidate masks and preserving image embedding features, thereby improving matching efficiency; it improves the targeting, accuracy, and stability of image structure matching by comparing structural information and calculating similarity through multi-feature combinations; and it solves the problem of accurately identifying ring targets when the segmentation results are incomplete by introducing a ring candidate reconstruction method. Therefore, the solution of embodiments of this disclosure has strong versatility and engineering application value, and can provide reliable technical support for applications such as defect detection, target localization, image alignment, structural analysis, and process judgment.

[0060] Figure 3 A schematic diagram of a system 300 for image recognition according to an embodiment of the present disclosure is shown. Figure 3 System 300 in the middle can be Figure 1 The system operates on electronic device 110 and any suitable electronic device. The system 300 includes: a candidate segmentation generation module 304, used to segment the target image to generate a mask set, wherein each mask in the mask set serves as a candidate mask, and the candidate masks are used to identify corresponding candidate regions in the target image; and a structure matching decision module 314, used to select a matching mask from the candidate masks that matches a reference mask, to obtain a target region in the target image that matches a reference region in the reference image identified by the reference mask.

[0061] Optionally, the system 300 may further include one or more of the following: an image input module 302 for acquiring a target image, a reference image, and a reference mask, wherein the candidate segmentation generation module 304 is connected to the image input module 302; an embedding retention module 306 for retaining the image embedding features of the target image, wherein the embedding retention module 306 is connected to the candidate segmentation generation module 304; a reference structure template extraction module 308 for extracting the structural information of the reference region identified by the reference mask in the reference image, wherein the reference structure template extraction module 308 is connected to both the image input module 302 and the embedding retention module 306; and a candidate selection module 310 for selecting candidates based on the structure of the reference region identified by the reference mask in the reference image. The system selects masks that meet predetermined conditions from candidate masks as new candidate masks. These new candidate masks serve as input to modules following the candidate selection module 310 for subsequent processing. The candidate selection module 310 is connected to the reference structure template extraction module 308. A similarity calculation module 312 determines the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region. The similarity calculation module 312 is connected to the candidate selection module 310, and the structure matching decision module 314 is connected to the similarity calculation module 312. A result output module 316 outputs a matching mask, which is used for at least one of the following: defect detection, target localization, image alignment, and structural analysis. The result output module 316 is connected to the structure matching decision module 314. In some embodiments, when the similarity scores of candidate masks are close, the structure matching decision module 314 can further determine the candidate region with an area closer to the reference region as the target region.

[0062] Figure 4 A flowchart of a method 400 for image recognition according to some embodiments of the present disclosure is shown. In some embodiments, the method performs image recognition based on segmentation candidates and multi-feature combinations.

[0063] Combining Figure 1 To describe method 400. Figure 4 Method 400 can be found in Figure 1 The steps are executed at electronic device 110 and any suitable electronic device. Furthermore, the numbers in the flowchart do not indicate the order in which these steps are executed; some or all of these steps may be executed in parallel, or their execution order may be interchanged, and this disclosure does not limit this.

[0064] The target image can be obtained within box 402. Reference image and reference mask .

[0065] Within box 404, the target image can be... Perform segmentation to generate a mask set , It can be represented as: (2) in, The number of masks generated. Indicates the first A mask is used. At the same time, the image embedding features of the target image are preserved in box 404.

[0066] Within box 406, structural information of the reference region identified by the reference mask in the reference image can be extracted. In some embodiments, the structural information may include at least one of the following: structure type, region area, center location, number of connected components, and aspect ratio.

[0067] In some embodiments, the area can be calculated using the following formula. : (3) Among them, when pixel When it belongs to the reference area, ,otherwise .

[0068] In some embodiments, the center position of the reference region can be calculated using image moments. Let the image moments be... If p and q are the orders of the image moments, then the coordinates of the target center are... It can be represented as: (4) in, This represents the zeroth moment of the reference mask. and They represent about and The first moment.

[0069] In box 408, candidate masks can be selected from the mask set based on structural information. In some embodiments, the center of the reference region is mapped onto the target image based on the center position obtained in box 406 to obtain the predicted center. Let the nth mask in the mask set be... The center coordinates of each mask are If the candidate region identified by the mask satisfies the following formula, then the mask is determined as a candidate mask: (5) in, It is the predefined radius of the region of interest. By constraining the region of interest using formula (5), candidate masks with a high degree of matching with the location of the reference region can be selected. In some embodiments, the range of candidate masks can be further narrowed by combining conditions such as area range, aspect ratio range, structural type consistency, and the number of connected components.

[0070] In decision box 410, it is determined whether the reference region is a solid structure. If the reference region is not a solid structure, method 400 proceeds to decision box 412. In decision box 412, it is determined whether the candidate region is a ring structure.

[0071] The following will refer to Figure 5 Describe in detail the specific process of the method for determining the structure type.

[0072] In box 412, if the candidate region is determined to be a circular structure, method 400 proceeds to box 416 and calculates the similarity. If the candidate region is determined not to be a circular structure, method 400 proceeds to box 414 and performs circular candidate reconstruction. Circular candidate reconstruction refers to selecting a first candidate mask and a second candidate mask from the candidate masks corresponding to candidate regions with a circular structure type in the candidate mask set. The area of ​​the first candidate mask is smaller than that of the second candidate mask, and the shape of the first candidate mask is contained within the shape of the second candidate mask. By subtracting the first candidate mask from the second candidate mask, a reconstructed candidate mask is generated as a new candidate mask.

[0073] In some embodiments, the candidate masks corresponding to all candidate regions with a ring-shaped structure are traversed. For any two candidate masks, it is determined whether there is an inclusion relationship between the candidate regions they identify. If there is no inclusion relationship, the other candidate mask combinations are evaluated; if there is an inclusion relationship, the candidate mask with the smaller area is selected as the first candidate mask, and the candidate mask with the larger area is selected as the second candidate mask. In some embodiments, the first candidate mask is set as... The second candidate mask is Then the candidate mask for reconstruction can be expressed as: (6) in, Indicates the first candidate mask The supplement to .

[0074] Then, method 400 proceeds to box 416, where a new candidate mask is computed. Similarity to the reference mask.

[0075] If the reference region is determined to be a solid structure in decision box 410, method 400 proceeds from decision box 410 to box 416 and calculates the similarity. In some embodiments, the image embedding similarity can be calculated using the following formula. : (7) in, Indicates the embedding features of the reference region. Indicates the first Embedding features of candidate regions identified by candidate masks.

[0076] In some embodiments, the grayscale image is set as follows: The grayscale range is [0, 255], and the histogram binning number is [number missing]. Then the normalized grayscale histogram of the reference region. ( Indicates by Composed of real numbers The first feature space) b Each component for: (8) in, It is a set of pixels in the reference region; It is the total number of pixels in the pixel set; This is an indicator function; its value is 1 if the equation is true, and 0 otherwise. This indicates rounding down, thus mapping grayscale values ​​to histograms.

[0077] Normalized gray-level histogram of the candidate region identified by the nth candidate mask No. b Each component for: (9) in, It is a set of candidate region pixels. It is the total number of pixels in the pixel set.

[0078] The following formula can be used to calculate the similarity of grayscale distributions: (10) (11) in, express and The distance of the Bach, This indicates the similarity of grayscale distributions.

[0079] In some embodiments, the boundary pixel set is extracted to the binary mask. Perform morphological erosion on the boundary pixel set. It can be defined as: (12)

[0080] in, This indicates the use of 3×3 rectangular structural elements. Perform corrosion operation; The logical operator "AND" requires that both sides of the equation be true simultaneously.

[0081] Let the grayscale image be Each boundary pixel gradient components on the x and y axes , It can be represented as: (13)

[0082] Then gradient magnitude It can be represented as: (14).

[0083] gradient direction It can be represented as: (15) in It is a two-parameter arctangent function.

[0084] Let the number of bins in the histogram of gradient directions be... Then the boundary gradient direction histogram of the reference region ( Indicates by Composed of real numbers The first feature space) d Each component can be represented as: (16) in, This represents the set of boundary pixels of the reference region. It is the sum of the magnitudes of the boundary gradients.

[0085] Histogram of gradient directions at the boundary of candidate regions The d Each component can be represented as: (17)

[0086] in, Represents the set of boundary pixels of the candidate region. It is the sum of the magnitudes of the boundary gradients.

[0087] The similarity of boundary gradient directions can then be calculated using the following formula: (18) (19) in, This represents the total original overlap between the candidate region and the reference region in the distribution of the boundary gradient direction. Indicates the first k Similarity of boundary gradient directions among candidate regions.

[0088] Then the first k The similarity of candidate regions can be calculated using the following formula: (20) in, , and The weighting coefficients for the corresponding similarity.

[0089] In some embodiments, similarity may also be a weighted sum of any two of image embedding similarity, grayscale distribution similarity, and boundary gradient direction similarity.

[0090] Then, method 400 proceeds to box 418 and determines the final matching result. In some embodiments, the candidate mask with the highest similarity can be determined as the matching mask. In some embodiments, the decision function value can be calculated using similarity and the area similarity between the candidate region and the reference region (as described in formula (1) above), and the candidate mask with the largest decision function value is determined as the matching mask, then the candidate index corresponding to the matching mask is determined. It can be represented as: (twenty one) in Indicates that the function The independent variable when it reaches its maximum value The value of .

[0091] Then, method 400 advances to box 420 and outputs the matching mask, that is, the output matching mask. It can be represented as: (twenty two)

[0092] In some embodiments, if the similarity of all candidate masks is lower than a preset similarity threshold, it is considered that there is no reliable target region in the target image corresponding to the reference region, and the method will output an empty result.

[0093] Figure 5A flowchart of a method 500 for determining the structure type according to an embodiment of the present disclosure is shown. Figure 5 The process can be found in Figure 1 The steps are executed at electronic device 110 and any suitable electronic device. Furthermore, the numbers in the flowchart do not indicate the order in which these steps are executed; some or all of these steps may be executed in parallel, or their execution order may be interchanged, and this disclosure does not limit this.

[0094] Method 500 performs a structural determination on the mask to be processed, which includes the candidate mask and the reference mask mentioned above. In block 502, multiple contours can be extracted from the mask to be processed based on the contour hierarchy to obtain a set of contours with nested relationships between different levels.

[0095] In decision box 504, the contour set can be inspected to determine whether there are holes inside the mask based on the nesting relationship between the layers; that is, to detect whether there is a contour hierarchy relationship between any two contours in the contour set. If there are holes inside the mask (i.e., there is a contour hierarchy relationship), method 500 proceeds to box 506 and determines the area identified by the mask to be processed as a ring structure. If there are no holes inside the mask (i.e., there is no contour hierarchy relationship), method 500 proceeds to box 508.

[0096] In box 508, the contour with the largest area among multiple contours can be selected as the largest contour, and its roundness index can be calculated. In some embodiments, the roundness index can be calculated using the following formula: (twenty two)

[0097] in, The roundness index representing the profile. This represents the area of ​​the outline. This indicates the perimeter of the outline.

[0098] Then, method 500 proceeds to decision box 510. In box 510, it is determined whether the roundness index calculated in box 508 is lower than a preset threshold. If the roundness index is lower than the preset threshold, method 500 proceeds to box 512 and determines the region identified by the mask to be processed as a ring candidate structure. If the roundness index is not lower than the preset threshold, method 500 proceeds to box 514 and determines the region identified by the mask to be processed as a solid structure.

[0099] Figure 6 A schematic diagram 600 illustrating an example of similarity calculation according to an embodiment of the present disclosure is shown. For example... Figure 6 As shown, the reference mask is The target image was segmented to generate four masks, namely... , , , Based on the structural information, the reference mask... The marked area is a ring structure, while With reference mask The structural types of the identified areas are clearly inconsistent, therefore, [they were selected]. , , As a candidate mask, the similarity between the candidate mask and the reference mask can be calculated separately, as described above. For example, the candidate mask... With reference mask Image embedding similarity The similarity of grayscale distribution is 0.92. The similarity of the boundary gradient directions is 0.88. The value is 0.90. In this example, let the weighting coefficients be respectively... , as well as Then the candidate mask With reference mask The similarity is Similarly, it can be calculated that... , and similarity , The scores were 0.641 and 0.500, respectively. This is based on the similarity comparison results of the candidate masks. , candidate mask It has been determined to be a matching mask.

[0100] Therefore, the solution of this disclosure, by segmenting candidate masks and preserving image embedding features, saves the time and computational cost caused by repeatedly performing feature encoding forward propagation on the same target image, thus improving matching efficiency. Through similarity calculation using multi-feature combinations, it considers both high-level semantics and low-level details of the image, improving the robustness of image matching in complex scenarios with deformation, defects, and noise. Addressing the problem of hollow or nested target images being prone to breakage and splitting during automatic segmentation, a ring candidate reconstruction method is introduced, effectively overcoming the difficulty in accurately identifying ring targets when the segmentation results are incomplete, and improving the recognition capability of complex topological structures. The introduction of area similarity effectively suppresses mismatches caused by local texture similarity but inconsistent overall structure, or by accidental high scores in boundary noise, when relying on other similarities for matching, thus improving the reliability and usability of image recognition. Therefore, this disclosure achieves high-precision, low-latency, and robust recognition and matching in complex scenarios of high resolution, high noise, and multiple candidate masks in semiconductor manufacturing.

[0101] Figure 7A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. Device 700 can be used to implement... Figure 1 The electronic device 110. As shown, the device 700 includes a processing unit 701, such as a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0102] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processing unit 701 executes the various methods and processes described above, such as methods 200, 400, and 500. For example, in some embodiments, methods 200, 400, and 500 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by processing unit 701, one or more steps of methods 200, 400, and 500 described above may be performed. Alternatively, in other embodiments, processing unit 701 may be configured to execute methods 200, 400, and 500 by any other suitable means (e.g., by means of firmware).

[0104] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0105] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0108] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for image recognition, comprising: The target image is segmented to generate a mask set, wherein each mask in the mask set is used as a candidate mask, and the candidate mask is used to identify the corresponding candidate region in the target image; as well as Using an image-based matching method, a matching mask that matches the reference mask is selected from the candidate masks to obtain the target region in the target image that matches the reference region identified by the reference mask in the reference image.

2. The method of claim 1, wherein using an image-information-based matching method to select a matching mask from the candidate masks that matches the reference mask, to obtain a target region in the target image that matches a reference region in the reference image identified by the reference mask, comprises: Based on the comparison between the structural information of the reference region and the structural information of the candidate region, the similarity between the candidate mask and the reference mask is determined. as well as The candidate mask with the highest similarity among the candidate masks is determined as the matching mask.

3. The method of claim 2, wherein determining the similarity between the candidate mask and the reference mask comprises: Determine a weighted sum of at least two of the following: the image embedding similarity between the candidate mask and the reference mask, the grayscale distribution similarity between the candidate mask and the reference mask, and the boundary gradient direction similarity between the candidate mask and the reference mask.

4. The method according to claim 3, further comprising: When segmenting the target image, the image embedding features of the target image are preserved; Based on the candidate mask, the image embedding features of the target image are subjected to region pooling to generate a local feature vector corresponding to the candidate mask; and The image embedding similarity is calculated using the local feature vector and the image embedding features of the reference region.

5. The method according to any one of claims 2-4, wherein the structural information includes at least one of the following: structural type, region area, center location, number of connected components, and aspect ratio.

6. The method according to claim 2, wherein determining the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region includes: Based on the fact that the structure of the reference region is a solid structure and the structure of the candidate region is a solid structure, the similarity between the candidate mask and the reference mask is calculated; Based on the fact that the reference region has a ring structure and the candidate region has a ring structure, the similarity between the candidate mask and the reference mask is calculated; as well as Based on the fact that the reference region has a ring structure and the candidate region has a ring candidate structure, ring candidate reconstruction is performed on the candidate mask to generate a reconstruction candidate mask, and the similarity between the reconstruction candidate mask and the reference mask is calculated.

7. The method of claim 6, wherein performing circular candidate reconstruction on the candidate mask to generate a reconstruction candidate mask, and calculating the similarity between the reconstruction candidate mask and the reference mask includes: A first candidate mask and a second candidate mask are selected from the candidate masks corresponding to the candidate regions that are determined to be ring-shaped candidate structures, wherein the area of ​​the first candidate mask is smaller than that of the second candidate mask and the pattern of the first candidate mask is contained by the pattern of the second candidate mask; as well as By subtracting the first candidate mask from the second candidate mask, a reconstructed candidate mask is generated as a new candidate mask, and the similarity between the new candidate mask and the reference mask is calculated.

8. The method according to claim 2, wherein determining the similarity between the candidate mask and the reference mask based on a comparison of the structural information of the reference region and the structural information of the candidate region comprises: Multiple contours are extracted from the mask to be processed, the mask to be processed including the candidate mask and the reference mask; as well as When there is a contour hierarchy among the multiple contours, the structure of the region identified by the mask to be processed is determined to be a ring structure; when there is no contour hierarchy among the multiple contours, the contour with the largest area among the multiple contours is selected as the largest contour.

9. The method of claim 8, further comprising: Calculate the roundness of the maximum contour; as well as When the roundness of the maximum contour is lower than a preset threshold, the structure of the region identified by the mask to be processed is determined to be a ring candidate structure; when the roundness of the maximum contour is not lower than the preset threshold, the structure of the region identified by the mask to be processed is determined to be a solid structure.

10. The method of claim 2, wherein determining the candidate mask with the highest similarity among the candidate masks as the matching mask comprises: The decision function value is calculated using the aforementioned similarity and area similarity. as well as The candidate mask with the largest decision function value is determined as the matching mask.

11. The method of claim 10, wherein the decision function value is calculated using the following formula: in, Let k be the decision function of the candidate mask. This indicates the similarity between the candidate mask k and the reference mask. This represents the area of ​​the candidate mask k. This represents the area of ​​the reference mask. The weighting coefficients representing the similarity and The weighting coefficient represents the area similarity.

12. The method of claim 1, wherein the matching mask is applied to at least one of: defect detection, target localization, image alignment, and structural analysis.

13. A system for image recognition, comprising: A candidate segmentation generation module is used to segment a target image to generate a mask set, wherein each mask in the mask set serves as a candidate mask, and the candidate masks are used to identify corresponding candidate regions in the target image. The structure matching decision module is used to select a matching mask that matches the reference mask from the candidate masks, so as to obtain the target region in the target image that matches the reference region in the reference image identified by the reference mask.

14. The system of claim 13, further comprising: An image input module is used to acquire the target image, the reference image, and the reference mask.

15. The system of claim 13, further comprising: An embedding retention module is used to retain the image embedding features of the target image.

16. The system of claim 13, further comprising: The reference structure template extraction module is used to extract the structural information of the reference region identified by the reference mask in the reference image.

17. The system of claim 13, further comprising: The similarity calculation module is used to determine the similarity between the candidate mask and the reference mask based on the comparison between the structural information of the reference region and the structural information of the candidate region.

18. The system of claim 13, further comprising: The result output module is used to output the matching mask, which is used for at least one of the following: defect detection, target localization, image alignment, and structural analysis.

19. An electronic device comprising: One or more processors; as well as A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the method according to any one of claims 1-12.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1-12.

21. A computer program product comprising program code that, when executed by a processor, implements the method according to any one of claims 1-12.

Images