Image matching method and device, equipment, storage medium and program product

Abstract

The invention provides an image matching method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a defect image and a target image; the scale of the defect image is different from that of the target image; extracting a foreground part of the defect image and a foreground part of the target image, and taking the foreground part of the defect image as a template image and the foreground part of the target image as an image to be matched; according to a matching relationship between a first feature vector corresponding to the template image and a second feature vector corresponding to the to-be-matched image, aligning the template image with the to-be-matched image under the same target scale; and determining actual coordinate information of the defect image in the target image according to the aligned template image and the to-be-matched image. According to the image matching method and device, the template images of different scales can be aligned with the to-be-matched image under the target scale, the position of the defect image in the target image is finally determined, and the flexibility and accuracy of image matching are improved.

Description

Technical Field

[0001] This application relates to the field of integrated circuit photolithography processes, and more particularly to an image matching method, apparatus, device, storage medium, and program product. Background Technology

[0002] Template matching is a technique widely used in computer vision and image processing, primarily used to find parts of an image that match a predefined template.

[0003] In related technologies, template matching is usually performed on images of the same scale. However, since the scale of images in real-world scenarios is usually different, the image matching methods in related technologies are not very flexible and have low accuracy when the image scales are different. Summary of the Invention

[0004] This application provides an image matching method, apparatus, device, storage medium, and program product, which can improve the flexibility and accuracy of image matching.

[0005] In a first aspect, embodiments of this application provide an image matching method, including:

[0006] Acquire a defect image and a target image; the scale of the defect image is different from the scale of the target image;

[0007] Extract the foreground portion of the defective image and the foreground portion of the target image, and use the foreground portion of the defective image as the template image and the foreground portion of the target image as the image to be matched;

[0008] Based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched, the template image and the image to be matched are aligned at the same target scale.

[0009] Based on the aligned template image and the image to be matched, the actual coordinate information of the defective image in the target image is determined.

[0010] In one possible implementation, extracting the foreground portion of the defective image and the foreground portion of the target image, using the foreground portion of the defective image as a template image and the foreground portion of the target image as an image to be matched, includes:

[0011] The defect image and the target image are respectively input into the target semantic segmentation network model;

[0012] The foreground portion of the defective image and the foreground portion of the target image are extracted using the target semantic segmentation network model. The foreground portion of the defective image is used as the template image and the foreground portion of the target image is used as the image to be matched.

[0013] In one possible implementation, the method further includes:

[0014] Obtain a sample dataset and divide the sample dataset into a training dataset, a validation dataset, and a test dataset according to a preset ratio;

[0015] Based on the training dataset, validation dataset, and test dataset, a preset semantic segmentation network model is trained and a predicted image is output.

[0016] Based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated through a preset optimizer to obtain the target semantic segmentation network model.

[0017] In one possible implementation, the method further includes:

[0018] Based on a preset feature extraction model, the first feature point of the template image is extracted and the first feature vector corresponding to the first feature point is determined.

[0019] Based on the preset feature extraction model, the second feature points of the image to be matched are extracted and the second feature vector corresponding to the second feature points is determined.

[0020] In one possible implementation, aligning the template image and the image to be matched at the same target scale based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched includes:

[0021] Based on the distance between the first feature vector and the second feature vector, the first feature vector and the second feature vector are matched to obtain an intermediate feature vector pair;

[0022] The intermediate feature vector pairs are filtered to obtain the target feature vector pairs;

[0023] Calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector pairs;

[0024] The template image is subjected to an affine transformation based on the affine transformation matrix, so that the template image and the image to be matched are aligned at the target scale; the target scale includes the target size and the target rotation angle.

[0025] In one possible implementation, determining the actual coordinate information of the defective image in the target image based on the aligned template image and the image to be matched includes:

[0026] If the aligned template image and the image to be matched are successfully matched, the offset coordinates between the template image and the image to be matched are calculated based on the center point coordinates of the template image and the center point coordinates of the image to be matched.

[0027] The actual coordinates of the defective image in the target image are determined based on the actual center point coordinates of the target image and the offset coordinates.

[0028] Secondly, embodiments of this application provide an image matching device, comprising:

[0029] An acquisition module is used to acquire a defect image and a target image; the scale of the defect image is different from the scale of the target image.

[0030] An extraction module is used to extract the foreground portion of the defective image and the foreground portion of the target image, using the foreground portion of the defective image as a template image and the foreground portion of the target image as an image to be matched.

[0031] The alignment module is used to align the template image and the image to be matched at the same target scale based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched.

[0032] The determination module is used to determine the actual coordinate information of the defective image in the target image based on the aligned template image and the image to be matched.

[0033] In one possible implementation, the extraction module is specifically used for:

[0034] The defect image and the target image are respectively input into the target semantic segmentation network model;

[0035] The foreground portion of the defective image and the foreground portion of the target image are extracted using the target semantic segmentation network model. The foreground portion of the defective image is used as the template image and the foreground portion of the target image is used as the image to be matched.

[0036] In one possible implementation, the device is further used for:

[0037] Obtain a sample dataset and divide the sample dataset into a training dataset, a validation dataset, and a test dataset according to a preset ratio;

[0038] Based on the training dataset, validation dataset, and test dataset, a preset semantic segmentation network model is trained and a predicted image is output.

[0039] Based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated through a preset optimizer to obtain the target semantic segmentation network model.

[0040] In one possible implementation, the device is further used for:

[0041] Based on a preset feature extraction model, the first feature point of the template image is extracted and the first feature vector corresponding to the first feature point is determined.

[0042] Based on the preset feature extraction model, the second feature points of the image to be matched are extracted and the second feature vector corresponding to the second feature points is determined.

[0043] In one possible implementation, the alignment module is specifically used for:

[0044] Based on the distance between the first feature vector and the second feature vector, the first feature vector and the second feature vector are matched to obtain an intermediate feature vector pair;

[0045] The intermediate feature vector pairs are filtered to obtain the target feature vector pairs;

[0046] Calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector pairs;

[0047] The template image is subjected to an affine transformation based on the affine transformation matrix, so that the template image and the image to be matched are aligned at the target scale; the target scale includes the target size and the target rotation angle.

[0048] In one possible implementation, the determining module is specifically used for:

[0049] If the aligned template image and the image to be matched are successfully matched, the offset coordinates between the template image and the image to be matched are calculated based on the center point coordinates of the template image and the center point coordinates of the image to be matched.

[0050] The actual coordinates of the defective image in the target image are determined based on the actual center point coordinates of the target image and the offset coordinates.

[0051] Thirdly, embodiments of this application provide an image matching device, including: a processor and a memory;

[0052] The memory stores computer-executed instructions;

[0053] The processor executes computer execution instructions stored in the memory to implement the image matching method as described in any of the first aspects.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the image matching method described in any of the first aspects.

[0055] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed, implements the image matching method described in any of the first aspects.

[0056] The image matching method, apparatus, device, storage medium, and program product provided in this application acquire a defect image and a target image; the scale of the defect image and the scale of the target image are different; the foreground portion of the defect image and the foreground portion of the target image are extracted, and the foreground portion of the defect image is used as a template image and the foreground portion of the target image is used as the image to be matched; based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched, the template image and the image to be matched are aligned at the same target scale; based on the aligned template image and the image to be matched, the actual coordinate information of the defect image in the target image is determined. In this application, after acquiring a defect image and a target image with different scales, the electronic device extracts the foreground portion of the defect image as a template image and extracts the foreground portion of the target image as the image to be matched; then, based on the matching relationship between the feature vectors of the template image and the image to be matched, alignment processing is performed at the same target scale; finally, based on the aligned template image and the image to be matched, the actual coordinate information of the defect image in the target image is determined, thereby realizing the matching and positioning of the defect image in the target image. In this way, the electronic device in this application can align template images of different scales with the image to be matched at the target scale by extracting the foreground image and matching the feature vector, and finally determine the position of the defect image in the target image, thereby improving the flexibility and accuracy of image matching. Attached Figure Description

[0057] Figure 1 A flowchart illustrating an image matching method provided in an embodiment of this application;

[0058] Figure 2 A flowchart illustrating another image matching method provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram of a target image provided in an embodiment of this application;

[0060] Figure 4 This is a partial network architecture diagram of a target semantic segmentation network model provided in an embodiment of this application;

[0061] Figure 5 A schematic diagram of a defect image and a template image provided for an exemplary embodiment of this application;

[0062] Figure 6 A schematic diagram of feature vector matching based on the nearest neighbor matching algorithm provided in this application embodiment;

[0063] Figure 7 A schematic diagram of a matched template image and a matching image provided in an embodiment of this application;

[0064] Figure 8 This is a schematic diagram of the structure of an image matching device provided in an embodiment of this application;

[0065] Figure 9 This is a schematic diagram of the structure of an image matching device provided in an embodiment of this application. Detailed Implementation

[0066] To enable those skilled in the art to better understand the technical solutions of this application, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are merely for explaining this application and are not intended to limit this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0067] Template matching is a widely used technique in computer vision and image processing, primarily used to find regions in an image that match a predefined template. The specific process involves comparing a known template with sub-regions in the target image one by one, finding the region in the target image that is most similar to the template. Due to its intuitiveness and ease of use, template matching is widely used in tasks such as pattern recognition, object detection, and object tracking. The main advantage of template matching is its simplicity and intuitiveness. It does not rely on complex machine learning algorithms or training processes; given a template, it can search within an image. This makes template matching suitable for use in resource-constrained environments, such as embedded systems or scenarios requiring rapid prototyping.

[0068] Furthermore, template matching can handle a variety of tasks, such as object detection, image alignment, object localization, and character recognition. Specifically, object detection refers to the quality inspection process on a production line, where template matching can be used to identify defects in specific shapes or patterns to ensure product quality. Image alignment refers to the registration of multiple images, where template matching helps to accurately align images, a common practice in medical image analysis. Object localization refers to scenarios such as robot vision, where template matching can be used to determine the position of objects, thereby guiding robotic arms to perform accurate operations. Character recognition refers to the use of template matching in Optical Character Recognition (OCR) to identify and match characters in standard fonts.

[0069] Despite its numerous advantages, template matching also has some significant drawbacks that limit its application in complex scenarios. The main problems with template matching methods in related technologies are as follows:

[0070] First, template matching is highly sensitive to changes in image illumination, scale, rotation, and noise. Even slight image distortion can lead to matching failure. This necessitates strict control over image acquisition conditions in practical applications, or the use of multi-scale, multi-angle templates to address variations, which significantly increases computational costs. Second, template matching has high computational complexity, especially when matching large images, requiring substantial computational resources. This makes it unsuitable for applications with high real-time requirements. Furthermore, template matching methods are typically only applicable to the recognition of known shapes or patterns. For complex scenes or target recognition with high uncertainty, template matching performance may be unsatisfactory. Compared to methods like deep learning, template matching lacks adaptive learning capabilities and cannot effectively handle complex patterns and variations in images.

[0071] To address the aforementioned problems, this application provides an image matching method, apparatus, device, storage medium, and program product. After acquiring defect images and target images of different scales, the electronic device extracts the foreground portion of the defect image as a template image and extracts the foreground portion of the target image as the image to be matched. Then, based on the matching relationship of feature vectors between the template image and the image to be matched, alignment processing is performed at the same target scale. Finally, based on the aligned template image and the image to be matched, the actual coordinate information of the defect image in the target image is determined, achieving the matching and positioning of the defect image in the target image. Thus, the electronic device in this application, through foreground image extraction and feature vector matching, can align template images and images to be matched at different scales at the target scale, ultimately determining the position of the defect image in the target image, improving the flexibility and accuracy of image matching.

[0072] The following detailed description of the solution presented in this application is provided through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0073] Figure 1 This is a schematic flowchart illustrating an image matching method provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 The image matching method may include:

[0074] S101. Obtain the defect image and the target image; the scale of the defect image is different from the scale of the target image.

[0075] The execution subject of this application embodiment can be an electronic device or an image matching device installed in an electronic device. The image matching device can be implemented by software or by a combination of software and hardware. For ease of understanding, the following description uses an electronic device as the execution subject. The electronic device can specifically refer to a mobile phone, computer, or cloud, etc. The specific type of electronic device is not limited in this application embodiment.

[0076] In this embodiment, the defect image can refer to an image matching template, specifically a scanning electron microscope (SEM) image in a semiconductor manufacturing context. The target image can refer to the image matched with the image template in image matching, specifically a graphic data system (GDS) image in a semiconductor manufacturing context. Scale can refer to the size and / or rotation angle of the image.

[0077] Specifically, in semiconductor manufacturing scenarios, equipment can acquire defect images during the semiconductor manufacturing process through SEM image capture. Electronic equipment can acquire this defect image and simultaneously acquire the corresponding GDS image as the target image, performing a subsequent image matching process to quickly locate the position of the SEM image captured by the equipment within the GDS image, thus achieving rapid defect detection and localization. In the embodiments of this application, the defect image and the target image may have different scales, specifically different sizes, different rotation angles, or both different sizes and rotation angles.

[0078] S102. Extract the foreground portion of the defect image and the foreground portion of the target image, and use the foreground portion of the defect image as the template image and the foreground portion of the target image as the image to be matched.

[0079] In this embodiment, the template image can refer to the image corresponding to the foreground portion of the defective image. The image to be matched can refer to the image corresponding to the foreground portion of the target image.

[0080] In this step, after acquiring the defect image and the target image, the electronic device can first extract the foreground portion. This improves the accuracy of subsequent image matching and avoids interference from factors such as lighting and noise. Specifically, the electronic device can segment the defect image and the target image based on a pre-trained target semantic segmentation network model, extracting the foreground portions of both the defect image and the target image. Then, the electronic device can use the foreground portion of the defect image as the template image and the foreground portion of the target image as the image to be matched.

[0081] S103. Based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched, align the template image and the image to be matched at the same target scale.

[0082] In this embodiment, the first feature vector may refer to the image feature vector corresponding to the template image. The second feature vector may refer to the image feature vector corresponding to the image to be matched. The target scale may refer to the scale used for image alignment processing, and may specifically include the target size and target rotation angle, etc.

[0083] In this step, after acquiring the template image corresponding to the defect image and the matching image corresponding to the target image, the electronic device can extract the first feature vector corresponding to the template image and the second feature vector corresponding to the matching image. Then, it compares and matches these first and second feature vectors, aligning the template image and the matching image at the same target scale based on their matching relationship. Since the defect image and the target image are at different scales, the electronic device, through feature vector extraction, feature vector matching, and image alignment at the same target scale, can ensure that defect images and target images of different scales are aligned at the same scale, thereby improving the accuracy of image matching.

[0084] S104. Based on the aligned template image and the image to be matched, determine the actual coordinate information of the defect image in the target image.

[0085] In this embodiment, the actual coordinate information can refer to the position information of the defect image in the target image. Specifically, after aligning the template image with the image to be matched, the electronic device can perform template detection processing to further ensure the matching accuracy between the template image and the image to be matched; then, the electronic device can perform coordinate calculation on the template image and the image to be matched, and finally obtain the actual coordinate information of the defect image in the target image, realizing the rapid positioning of the defect image in the target image and improving the accuracy and flexibility of image matching.

[0086] The image matching method provided in this application involves an electronic device acquiring a defect image and a target image. The scales of the defect image and the target image are different. The foreground portions of both the defect image and the target image are extracted. The foreground portion of the defect image is used as a template image, and the foreground portion of the target image is used as the image to be matched. Based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched, the template image and the image to be matched are aligned at the same target scale. Based on the aligned template image and the image to be matched, the actual coordinate information of the defect image in the target image is determined. In this application, after acquiring defect images and target images of different scales, the electronic device extracts the foreground portion of the defect image as a template image and the foreground portion of the target image as the image to be matched. Then, based on the matching relationship between the feature vectors of the template image and the image to be matched, alignment processing is performed at the same target scale. Finally, based on the aligned template image and the image to be matched, the actual coordinate information of the defect image in the target image is determined, thus achieving the matching and positioning of the defect image in the target image. In this way, the electronic device in this application can align template images of different scales with the image to be matched at the target scale by extracting the foreground image and matching the feature vector, and finally determine the position of the defect image in the target image, thereby improving the flexibility and accuracy of image matching.

[0087] Based on the above embodiments, Figure 2 This is a schematic flowchart illustrating another image matching method provided in an embodiment of this application. Please refer to... Figure 2 The image matching method may include:

[0088] S201. Obtain the defect image and the target image; the scale of the defect image is different from the scale of the target image.

[0089] In this embodiment, the electronic device can specifically acquire a SEM image as a defect image and a GDS image as a target image. The GDS image can be used to describe the layout and circuit connections of a semiconductor chip, and can include information such as various geometric shapes, hierarchical structures, circuit elements, and interconnections. For example, Figure 3This is a schematic diagram of a target image provided in an embodiment of this application. For example... Figure 3 As shown, the target image of the GDS image includes multiple lines of different line widths and multiple circuit elements (not shown in the figure).

[0090] S202. Input the defect image and the target image into the target semantic segmentation network model respectively; through the target semantic segmentation network model, extract the foreground part of the defect image and the foreground part of the target image respectively, and use the foreground part of the defect image as the template image and the foreground part of the target image as the image to be matched.

[0091] In this embodiment, the target semantic segmentation network model can refer to a pre-trained multi-scale semantic segmentation network model. Based on the characteristic of varying linewidths in scanning electron microscope (SEM) images, the target semantic segmentation network model in this embodiment can be a multi-scale semantic segmentation convolutional neural network (CNN) model. This target semantic segmentation network model can employ a pyramid network (FPN) and residual resolution network (Res2Net) structure, combined with an attention mechanism (e.g., a squeeze-and-excitation block (SE block)). In this way, the target semantic segmentation network model can possess the ability to process multi-scale features of SEM images with different linewidths.

[0092] Specifically, in the target semantic segmentation network model, the pyramid network fuses features at different resolutions through upsampling and subsampling, enabling the fusion of multi-scale information between different layers. The residual resolution network structure further enhances the network's multi-scale information extraction capability by using different convolutional kernel sizes for different channels, achieving multi-scale information fusion between different receptive fields within the same layer. Furthermore, the target semantic segmentation network model introduces an SE Block channel attention mechanism, which effectively selects the most suitable receptive field size for the semantic segmentation task and significantly improves image segmentation accuracy by increasing the weight of the corresponding channel within that receptive field. Thus, through the pyramid network, residual resolution network, and attention mechanism, the target semantic segmentation network model can adapt to various scales and viewpoints of images, and through adaptive weight adjustment, further improve the robustness and accuracy of image segmentation, ensuring the accuracy of foreground extraction.

[0093] For example, Figure 4 This is a partial network architecture diagram of a target semantic segmentation network model provided in an embodiment of this application. Figure 4 As shown, the target semantic segmentation network model can include a residual resolution network Res2Net and an attention mechanism layer SE Block. The input image features first pass through a Conv1*1 convolutional layer, divided into four groups: X1, X2, X3, and X4. Except for the first group X1, the feature maps output by each group undergo a 3*3 convolution operation. Besides the first group X1 and the second group X2, the feature map of the i-th group (i=3 or 4) can be added to the output of the previous 3*3 convolution operation, and then subjected to another 3*3 convolution operation, resulting in four outputs: Y1, Y2, Y3, and Y4. These four outputs are then input to the attention mechanism layer, where weights are adjusted. The outputs of the attention mechanism layer are then concatenated along the channel dimension, and a Conv1*1 convolution operation is performed based on the convolutional layer to finally obtain the output image.

[0094] It should be noted that the above-described target semantic segmentation network model is merely an example. In this model, the CNN model can be replaced with other types of feature extraction networks, such as Vision Transformers (ViT), Multilayer Perceptron Mixers (MLP-Mixer), and next-generation convolutional networks (ConvNeXt). The FPN network can be replaced with other multi-scale feature fusion modules, such as Path Aggregation Networks (PAN), Bi-directional Feature Pyramid Networks (BiFPN), and High-Resolution Networks (HRNet). The residual resolution network can also employ other network structures that enhance feature representation, such as DenseNet, EfficientNet, and Inception modules. The SE Block attention mechanism can also be replaced with other attention mechanisms to enhance feature selection capabilities, such as Convolutional Block Attention Modules (CBAM), Efficient Channel Attention (ECA), and Non-Local Blocks. In the practical application of the target semantic segmentation network model, electronic devices can freely combine the above-mentioned different types of models, modules or networks, and the embodiments of this application do not limit this.

[0095] In this step, the electronic device can input the defect image and the target image into the target semantic segmentation network model, respectively. The target semantic segmentation network model can extract the foreground part of the defect image and the foreground part of the target image, respectively, and finally output the template image corresponding to the foreground part of the defect image and the image to be matched corresponding to the foreground part of the target image.

[0096] For example, Figure 5 This is a schematic diagram of a defect image and a template image provided for an exemplary embodiment of this application. Figure 5 In the image (a), the defect image is obtained by the electronic device. The device can then input this defect image into a target semantic segmentation network model. This model performs image segmentation on the defect image, extracting the foreground portion. Figure 5 Template image in (b).

[0097] In one possible implementation, the target semantic segmentation network model can be trained through the following steps (1) to (3):

[0098] (1) Obtain the sample dataset and divide the sample dataset into training dataset, validation dataset and test dataset according to the preset ratio.

[0099] In this embodiment, the sample dataset can refer to a pre-acquired set of sample data, which may include SEM image and GDS image pairs, and the foreground segmentation dataset corresponding to the image pairs. The preset ratio can refer to a pre-set specific ratio of the training dataset, validation dataset, and test dataset; for example, the preset ratio of the training dataset, validation dataset, and test dataset could be 6:2:2, etc. The training dataset can be used for iterative training of the model; the validation dataset can be used for initial model validation and parameter optimization; and the test dataset can be used for inference testing of the model.

[0100] (2) Based on the training dataset, validation dataset and test dataset, train the preset semantic segmentation network model and output the predicted image.

[0101] (3) Based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated by the preset optimizer to obtain the target semantic segmentation network model.

[0102] In this embodiment, the preset semantic segmentation network model can refer to an untrained semantic segmentation network model in its initial state. It can be built based on a preset framework, such as a deep learning framework (e.g., PyTorch). This preset semantic segmentation network model may include a pyramid network and a residual resolution network structure, and incorporates an attention mechanism. During image segmentation, deconvolution is used to restore the image scale, ensuring a one-to-one correspondence between input and output pixels. The predicted image can refer to the preliminary image segmentation result output by the preset semantic segmentation network model. The target loss function can refer to the loss function used during model training. The preset optimizer can be used to optimize and update the network parameters of the preset semantic segmentation network model.

[0103] Specifically, after dividing the sample dataset into training, validation, and test datasets, the electronic device can train a preset semantic segmentation network model based on the training, validation, and test datasets. For example, the training dataset can be input into the preset semantic segmentation network model to obtain a predicted image. Then, based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated using a preset optimizer to minimize the distance between the preset image and the foreground segmentation data in the sample dataset. Subsequently, validation and testing are performed based on the validation and test datasets respectively, and finally, the trained target semantic segmentation network model is obtained.

[0104] In this embodiment, the electronic device divides the sample dataset into training, validation, and test datasets. During training, it acquires a preset image output by a pre-defined semantic segmentation network model. Then, based on the preset image and the target loss function, it iteratively updates the network parameters using a pre-defined optimizer. After iterative training, the target semantic segmentation network model is obtained. In this way, the electronic device obtains the target semantic segmentation network model through training. Based on this model, it extracts the foreground portions of the defect image and the target image, avoiding noise interference and ensuring the accuracy of subsequent image matching.

[0105] S203. According to the preset feature extraction model, extract the first feature point of the template image and determine the first feature vector corresponding to the first feature point; according to the preset feature extraction model, extract the second feature point of the image to be matched and determine the second feature vector corresponding to the second feature point.

[0106] In this embodiment, the preset feature extraction model can refer to a pre-trained machine vision algorithm model, which can be used to extract key feature points and corresponding feature vectors from an image. Specifically, it can refer to a Scale-invariant feature transform (SIFT) model, a Speeded-Up Robust Features (SURF) model, or an Oriented Fast and Rotated BRIEF (ORB) model based on feature point detection and descriptor generation. This embodiment does not limit the specific type of the preset feature extraction model. The first feature point can refer to the key feature point corresponding to the template image; the second feature point can refer to the key feature point corresponding to the matching image.

[0107] In this step, after acquiring the template image and the image to be matched, the electronic device can first extract a first feature point from the template image based on a preset feature extraction model and determine the first feature vector corresponding to the first feature point. Simultaneously, the electronic device can extract a second feature point from the image to be matched based on the same scale-invariant feature transform model and determine the second feature vector corresponding to the second feature point. In this way, the electronic device extracts the first feature vector of the template image and the second feature vector of the image to be matched by applying the preset feature extraction model. The first and second feature points have strong scale and rotation invariance, maintaining consistency across images of different scales and ensuring the accuracy of subsequent feature vector matching.

[0108] S204. Match the first feature vector and the second feature vector according to the distance between the first feature vector and the second feature vector to obtain an intermediate feature vector pair; filter the intermediate feature vector pair to obtain the target feature vector pair.

[0109] In this embodiment, the intermediate feature vector pair can refer to the vector pair formed by the initial matching of the first feature vector and the second feature vector. The electronic device can determine the intermediate feature vector pair for the initial matching of the template image and the target image based on a preset feature matching algorithm, such as K-Nearest Neighbors (KNN), K-Means clustering, or Gaussian Mixture Model (GMM), according to the minimum distance between the first and second feature vectors. Of course, other types of algorithms can also be used for the preset feature matching algorithm, and the specific choice can be flexibly based on actual needs; this embodiment does not limit this choice.

[0110] For example, Figure 6This is a schematic diagram illustrating feature vector matching based on the nearest neighbor matching algorithm, provided as an embodiment of this application. Figure 6 As shown, for the first and second feature vectors, the K value in the KNN algorithm can be 2, and the matching coefficient can be 0.6. The electronic device performs feature matching based on this KNN algorithm, and can finally determine the feature points that are initially matched between the first and second feature vectors, thereby obtaining intermediate feature vector pairs. It should be noted that the electronic device can also use other feature matching algorithms for feature matching, and the specific requirements can be flexibly limited based on actual needs. This application embodiment does not limit this.

[0111] The target feature vector pair refers to the matching feature vector pair between the finally determined template image and the target image. After determining the intermediate feature vector pairs, the electronic device can filter these intermediate feature vector pairs, deleting those with incorrect matches, and finally obtaining the target feature vector pairs. Specifically, the electronic device can filter the intermediate feature vector pairs according to a preset filtering algorithm, such as Random Sample Consensus (RANSAC), Least Median of Squares (LMEDS), or Maximum Likelihood Estimation Sample Consensus (MLESAC). This preset filtering algorithm can be used to determine valid data from a dataset that includes erroneous data. Of course, other types of algorithms can also be used for the preset filtering algorithm, and the specific choice can be flexibly made based on actual needs. This application embodiment does not limit this.

[0112] In this step, after extracting the first and second feature vectors, the electronic device can first perform feature vector matching using a preset feature matching algorithm such as KNN to obtain intermediate feature vector pairs. Then, it can filter these intermediate feature vector pairs using a preset filtering algorithm such as random sampling consistency to finally obtain the target feature vector pairs between the template image and the image to be matched. In this way, the electronic device can improve the accuracy and rationality of feature matching based on feature matching and filtering, thereby improving the accuracy of subsequent image matching.

[0113] S205. Calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector pairs; perform an affine transformation on the template image based on the affine transformation matrix so that the template image and the image to be matched are aligned at the target scale; the target scale includes the target size and the target rotation angle.

[0114] In this embodiment, the affine transformation matrix can refer to the transformation matrix between two vector spaces, which may include a linear transformation and a translation transformation. The target size can refer to the size corresponding to the target scale, specifically the size of the image to be matched; the target rotation angle can refer to the image rotation angle corresponding to the target scale, specifically the rotation angle of the image to be matched.

[0115] Specifically, after determining the target matching vector pair between the template image and the image to be matched, the electronic device can iteratively select the target feature vector pair based on a preset filtering algorithm such as random sampling consistency. Then, it can calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector point pairs. The template image can then be affine transformed according to the affine transformation matrix, ultimately aligning the template image and the image to be matched at the target scale. This achieves alignment processing of images of different scales at the same size and rotation angle, ensuring the accuracy of subsequent coordinate calculations and thus improving the accuracy of image matching.

[0116] S206. If the aligned template image and the image to be matched are successfully matched, calculate the offset coordinates between the template image and the image to be matched based on the center point coordinates of the template image and the center point coordinates of the image to be matched.

[0117] S207. Determine the actual coordinate information of the defect image in the target image based on the actual center point coordinates and offset coordinates of the target image.

[0118] In this embodiment, after aligning the template image and the image to be matched, the electronic device can perform image matching between the template image and the image to be matched at the same target scale. If the matching is successful, the electronic device can perform subsequent coordinate calculation. If the image matching fails, the electronic device can determine that the template image and the image to be matched do not match, and there is no need to perform subsequent coordinate calculation. In this way, the electronic device can ensure that the template image and the target image are accurately matched at a uniform scale, thereby improving the reliability and accuracy of image matching.

[0119] Center point coordinates can refer to the coordinates of the center point of an image. When the aligned template image and the image to be matched are successfully matched, the coordinates of the center point of the template image become the coordinates of the center point of the matching box in the image to be matched. Offset coordinates can refer to the offset between the center point coordinates of the template image and the image to be matched. Specifically, the electronic device can determine the center point coordinates of the template image and the center point coordinates of the image to be matched after image matching. Based on these two center point coordinates, the electronic device can calculate the offset coordinates. Then, based on the actual center point coordinates of the target image and the offset coordinates, the actual coordinate information of the defective image in the target image can be calculated, thereby achieving the matching and positioning of the defective image in the target image.

[0120] For example, Figure 7 This is a schematic diagram illustrating a matched template image and a matching image provided in an embodiment of this application. For example... Figure 7 As shown, the template image 701 matches a portion of the image region in the image to be matched 702. After the image matching is successful, the electronic device can calculate the actual coordinate position of the defect image corresponding to the template image 701 in the target image corresponding to the image to be matched 702 based on the coordinates.

[0121] In the image matching method of this application embodiment, the electronic device extracts the foreground parts of the defect image and the target image as template images and images to be matched based on the target semantic segmentation network. Then, based on feature matching and alignment processing, the defect image is finally matched and located in the target image through coordinate calculation.

[0122] Traditional template matching is extremely sensitive to changes in image scale. When the template and target images are not at the same scale, the matching effect often drops significantly or even fails completely. Compared with traditional template matching methods, the image matching method in this application overcomes the limitation of image matching only at the same scale, and can achieve accurate matching across scales, handling the matching problem of defective images and target images at different scales.

[0123] The image matching method in this embodiment performs image segmentation based on a target semantic segmentation network model and image matching based on algorithms such as feature matching. This reduces interference from factors such as lighting and noise, and can adaptively adjust the rotation angle and scaling of the template image to fit the image to be matched, improving the flexibility and adaptability of the matching process. Furthermore, the image matching method in this embodiment has a wider range of applications and can be applied to more practical scenarios. For example, in semiconductor manufacturing, it can achieve rapid localization of SEM images within GDS images, significantly improving the detection accuracy and efficiency of defect images. The image matching method in this embodiment can meet diverse and complex practical application needs.

[0124] Figure 8 This is a schematic diagram of the structure of an image matching device provided in an embodiment of this application. Please refer to... Figure 8 The image matching device 80 may include:

[0125] The acquisition module 81 is used to acquire the defect image and the target image; the scale of the defect image is different from the scale of the target image.

[0126] Extraction module 82 is used to extract the foreground part of the defect image and the foreground part of the target image, using the foreground part of the defect image as the template image and the foreground part of the target image as the image to be matched.

[0127] Alignment module 83 is used to align the template image and the image to be matched at the same target scale based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched.

[0128] The determination module 84 is used to determine the actual coordinate information of the defect image in the target image based on the aligned template image and the image to be matched.

[0129] In one possible implementation, the extraction module 82 is specifically used for:

[0130] The defect image and the target image are respectively input into the target semantic segmentation network model;

[0131] Using a target semantic segmentation network model, the foreground parts of the defective image and the target image are extracted respectively. The foreground part of the defective image is used as the template image and the foreground part of the target image is used as the image to be matched.

[0132] In one possible implementation, the device 80 is further used for:

[0133] Obtain the sample dataset and divide it into training dataset, validation dataset, and test dataset according to a preset ratio;

[0134] Based on the training dataset, validation dataset, and test dataset, a pre-defined semantic segmentation network model is trained and a predicted image is output.

[0135] Based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated through a preset optimizer to obtain the target semantic segmentation network model.

[0136] In one possible implementation, the device 80 is further used for:

[0137] Based on the preset feature extraction model, the first feature point of the template image is extracted and the first feature vector corresponding to the first feature point is determined;

[0138] Based on the preset feature extraction model, the second feature points of the image to be matched are extracted and the second feature vector corresponding to the second feature points is determined.

[0139] In one possible implementation, the alignment module 83 is specifically used for:

[0140] Based on the distance between the first feature vector and the second feature vector, the first feature vector and the second feature vector are matched to obtain an intermediate feature vector pair;

[0141] The intermediate feature vector pairs are filtered to obtain the target feature vector pairs.

[0142] Calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector pairs;

[0143] An affine transformation is performed on the template image based on the affine transformation matrix, so that the template image and the image to be matched are aligned at the target scale; the target scale includes the target size and the target rotation angle.

[0144] In one possible implementation, the determining module 84 is specifically used for:

[0145] If the aligned template image and the image to be matched are successfully matched, the offset coordinates between the template image and the image to be matched are calculated based on the center point coordinates of the template image and the center point coordinates of the image to be matched.

[0146] Based on the actual center point coordinates and offset coordinates of the target image, determine the actual coordinate information of the defect image in the target image.

[0147] The image matching device 80 provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0148] Figure 9 This is a schematic diagram of the structure of an image matching device provided in an embodiment of this application. Please refer to... Figure 9 The image matching device 90 may include a memory 91 and a processor 92. Exemplarily, the memory 91 and the processor 92 are interconnected via a bus 93.

[0149] Memory 91 is used to store program instructions;

[0150] The processor 92 is used to execute the program instructions stored in the memory to implement the image matching method shown in the above embodiment.

[0151] Figure 9The image matching device 90 shown can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0152] This application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-described image matching method when executed by a processor.

[0153] This application embodiment may also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described image matching method.

[0154] This application provides a chip that stores a computer program. When the computer program is executed by the chip, the above-described image matching method is implemented.

[0155] This application provides a chip module that stores a computer program. When the computer program is executed by the chip module, the above-mentioned image matching method is implemented.

[0156] It should be noted that the processor mentioned in the embodiments of this application may be a central processing unit (CPU) or the like, and the memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The embodiments of this application do not limit the specific types of processors and memory.

[0157] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. Each device and product can be applied to or integrated into a chip, chip module, or terminal device. For example, for devices and products applied to or integrated into a chip, each included module / chip can be implemented entirely using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining modules / units can be implemented using hardware methods such as circuits.

[0162] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0163] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image matching method, characterized in that, include: Acquire defect images and target images; The scale of the defective image is different from the scale of the target image; Extract the foreground portion of the defective image and the foreground portion of the target image, and use the foreground portion of the defective image as the template image and the foreground portion of the target image as the image to be matched; Based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched, the template image and the image to be matched are aligned at the same target scale. Based on the aligned template image and the image to be matched, the actual coordinate information of the defective image in the target image is determined.

2. The method according to claim 1, characterized in that, The step of extracting the foreground portion of the defective image and the foreground portion of the target image, using the foreground portion of the defective image as a template image and the foreground portion of the target image as the image to be matched, includes: The defect image and the target image are respectively input into the target semantic segmentation network model; The foreground portion of the defective image and the foreground portion of the target image are extracted using the target semantic segmentation network model. The foreground portion of the defective image is used as the template image and the foreground portion of the target image is used as the image to be matched.

3. The method according to claim 2, characterized in that, The method further includes: Obtain a sample dataset and divide the sample dataset into a training dataset, a validation dataset, and a test dataset according to a preset ratio; Based on the training dataset, validation dataset, and test dataset, a preset semantic segmentation network model is trained and a predicted image is output. Based on the predicted image and the target loss function, the network parameters of the preset semantic segmentation network model are iteratively updated through a preset optimizer to obtain the target semantic segmentation network model.

4. The method according to claim 1, characterized in that, The method further includes: Based on a preset feature extraction model, the first feature point of the template image is extracted and the first feature vector corresponding to the first feature point is determined. Based on the preset feature extraction model, the second feature points of the image to be matched are extracted and the second feature vector corresponding to the second feature points is determined.

5. The method according to claim 1, characterized in that, The step of aligning the template image and the image to be matched at the same target scale based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched includes: Based on the distance between the first feature vector and the second feature vector, the first feature vector and the second feature vector are matched to obtain an intermediate feature vector pair; The intermediate feature vector pairs are filtered to obtain the target feature vector pairs; Calculate the affine transformation matrix based on the feature point pairs corresponding to the target feature vector pairs; The template image is subjected to an affine transformation based on the affine transformation matrix, so that the template image and the image to be matched are aligned at the target scale; the target scale includes the target size and the target rotation angle.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the actual coordinate information of the defective image in the target image based on the aligned template image and the image to be matched includes: If the aligned template image and the image to be matched are successfully matched, the offset coordinates between the template image and the image to be matched are calculated based on the center point coordinates of the template image and the center point coordinates of the image to be matched. The actual coordinates of the defective image in the target image are determined based on the actual center point coordinates of the target image and the offset coordinates.

7. An image matching device, characterized in that, include: The acquisition module is used to acquire defect images and target images; The scale of the defective image is different from the scale of the target image; An extraction module is used to extract the foreground portion of the defective image and the foreground portion of the target image, using the foreground portion of the defective image as a template image and the foreground portion of the target image as an image to be matched. The alignment module is used to align the template image and the image to be matched at the same target scale based on the matching relationship between the first feature vector corresponding to the template image and the second feature vector corresponding to the image to be matched. The determination module is used to determine the actual coordinate information of the defective image in the target image based on the aligned template image and the image to be matched.

8. An image matching device, characterized in that, include: Processor, memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the image matching method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the image matching method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed, implements the image matching method according to any one of claims 1 to 6.

Images