A circuit pattern feature matching method based on dense matching

Abstract

The application belongs to the technical field of image processing, and particularly relates to a circuit pattern feature matching method based on dense matching. The method firstly performs multi-scale feature fusion on a to-be-detected image and a template image respectively to obtain a low-resolution fusion image, then the multi-scale features of the two images are subjected to a convolution to obtain key point heat; the low-resolution fusion image is subjected to a linear layer to obtain a confidence map, finally in a feature matching stage, a feature descriptor is formed by key points with high confidence scores, the feature descriptor is adjusted to obtain an image descriptor, the similarity between the image descriptors is calculated to perform nearest neighbor matching, and a rough key point matching pair is calculated; the matching result obtained from the rough granularity matching stage is used for further fine alignment, the positions of the corresponding key points in the image are more accurately adjusted, and therefore more accurate matching is obtained. The method makes the positions of the matching key points more accurate, and realizes more accurate and efficient image matching.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically a circuit pattern feature matching method based on dense matching. Background Technology

[0002] Circuit pattern feature matching is a key quality control technique aimed at improving detection accuracy and efficiency through high-precision image processing methods. With advancements in deep learning, modern circuit pattern feature matching typically relies on building deep learning models to learn and understand the complex structures on circuit boards. However, because actual photographed circuit patterns often contain numerous subtle variations, traditional image processing techniques often struggle to fully capture and process this detailed information.

[0003] The dense matching-based approach overcomes the limitations of traditional matching methods by introducing the advanced XFeat model for feature extraction and matching. The XFeat model can not only effectively capture minute details and structural features in circuit graphics images, but also utilize the self-attention and cross-attention mechanisms of Transformer for refined feature matching and correction, thereby improving accuracy and stability.

[0004] In particular, the deep learning-based dense matching method can comprehensively consider image changes under different angles and lighting conditions when processing circuit graphics images, effectively improving the detection capability of complex defects and details. This method does not rely solely on traditional image analysis techniques, but combines advanced neural network architecture and large-scale data training, providing a more reliable and efficient quality control solution for the microelectronics manufacturing industry.

[0005] However, processing large circuit diagrams presents several challenges: 1) High computational resource requirements: Large circuit diagrams are high-resolution, demanding significant computational resources and memory for processing. 2) The circuit diagram data contains numerous similar elements, rendering traditional feature matching methods such as SIFT and ORB ineffective. 3) Complexity of feature extraction: Extracting effective features from large images poses a challenge for traditional algorithms and some deep learning models. Large images may contain numerous complex details and noise, requiring effective algorithms to extract and process these features. Summary of the Invention

[0006] To address the issues of high computational resource requirements and poor feature extraction performance when performing circuit pattern feature matching on large circuit diagrams using deep learning-based dense matching methods, this invention proposes a circuit pattern feature matching method based on dense matching, aiming to achieve more accurate and efficient image matching.

[0007] This invention is achieved using the following technical solution: a circuit pattern feature matching method based on dense matching, comprising the following steps:

[0008] Step S1, Multi-scale feature fusion:

[0009] Image to be detected With template image Multi-scale features were extracted from each. , … ... Ultimately, in multi-scale features The fusion process is performed to obtain the low-resolution fused image to be detected. Template low-resolution fusion image ;

[0010] Step S2, Key Point Detection:

[0011] Image to be detected With template image Multiscale features Each element undergoes a convolution, and is then flattened along the channel dimension to obtain a heatmap of the key points to be detected. Template key point heatmap ;

[0012] Step S3: Extract the confidence map:

[0013] For the low-resolution fusion image to be detected obtained in step S1 Template low-resolution fusion image Each of these layers passes through a linear layer to obtain the confidence map of the target. Template confidence plot This is used to evaluate which key points are effective.

[0014] Step S4, Feature Matching Stage:

[0015] Step S4-1, Coarse-grained matching stage:

[0016] Heat map of key points to be detected and the confidence graph of the target Combined template key point heatmap Template confidence plot By combining the confidence scores calculated for each keypoint separately, the top [keypoints] are selected. Keypoints with high confidence scores are identified, and interpolation methods are used to extract these keypoints to form feature descriptors. The extracted feature descriptors are then normalized, and finally, the normalized feature descriptors are adjusted according to the image scaling ratio to restore the resolution to multi-scale features. The resolution, i.e., the descriptor of the image to be detected. Template image descriptor ;

[0017] Calculate the image descriptor to be detected Template image descriptor The similarity between them is used to perform nearest neighbor matching, and a rough key point matching pair is calculated.

[0018] Step S4-2, Fine-grained matching stage:

[0019] Fine-grained alignment: In this stage, the matching results obtained from the coarse-grained matching stage are used to further refine the alignment, and the positions of corresponding key points in the image are adjusted more precisely to obtain a more accurate match.

[0020] In the aforementioned circuit pattern feature matching method based on dense matching, step S1 involves multi-scale features... … Using a combination of upsampling and 1×1 convolution, multi-scale features are made available. … Resolution fusion into multi-scale features The resolution, for multi-scale features … Then, DW convolution is used to combine multi-scale features. … Resolution reduced to multi-scale features The resolution, and finally the multi-scale features , … … The images are stitched together to obtain a low-resolution fused image with fused features.

[0021] In the aforementioned circuit pattern feature matching method based on dense matching, step S2 involves multi-scale features. After convolution processing, the number of channels increases by 1. The extra channel is designed as a dustbin channel to represent regions where no keypoints were detected.

[0022] In the aforementioned circuit pattern feature matching method based on dense matching, in step S4-1, firstly, the image descriptor to be detected... Template image descriptor Performing a dot product yields a similarity matrix for the image descriptors. The elements in the similarity matrix represent the similarity scores for each pair of keypoints. , , This represents an element in the similarity matrix of the image descriptors to be detected. Represents the elements in the similarity matrix of template image descriptors;

[0023] The maximum match of image descriptors is selected from the similarity matrix, i.e., the most similar keypoints are found, and then a mutual matching strategy is executed. , This yields keypoint matching pairs ( ), and These are the image descriptors to be detected. Key points and template image descriptors in The key points in In template image descriptor Zhongyu It is the most similar, and In the image descriptor to be detected Zhongyu They are the most similar.

[0024] The aforementioned circuit pattern feature matching method based on dense matching, in step S4-2, maps the matched key points on the low-resolution image descriptor back to the corresponding regions of the high-resolution original image based on the results of coarse-grained matching, thus mapping the image to be detected... With template image The corresponding regions are unfolded into sequences, and the features obtained through self-attention and cross-attention are then reassembled into a feature map; the image to be detected Feature maps and template images Perform point-by-point correlation operations on the feature map to obtain a correlation map, and then use the correlation map to extract the template image. Selecting from the feature map of the image to be detected Feature map center features Points with the highest cross-correlation are considered matched points, thus completing fine-grained matching; finally, the image to be detected is matched based on the results of fine-grained matching. With template image .

[0025] The aforementioned circuit pattern feature matching method based on dense matching is suitable for obtaining a low-resolution fused image to be detected. Low-resolution fusion image with template Use the Dual-softmax loss function. In the formula, This is a similarity matrix. , Indicates the low-resolution fused image to be detected The Middle Key points and template low-resolution fusion map The Middle The most similar key points Matching consistency is enhanced by jointly optimizing the log probabilities in two directions: one term maximizes the log probabilities from the low-resolution fusion graph to be detected. Low-resolution fusion image of template Matching probability Another factor is maximizing the low-resolution fusion image from the template. Back to the low-resolution fusion image to be detected Matching probability This bidirectional constraint mechanism not only forces correctly matched keypoints to be nearest neighbors in the feature space, satisfying the cyclic consistency constraint, but also effectively suppresses the response values ​​of non-matched keypoints, thereby significantly improving the discriminative ability of image descriptors in complex circuit textures.

[0026] The aforementioned circuit graph feature matching method based on dense matching designs a reliability loss function for the confidence graph. ,in, For the sigmoid function, For Hadamard products, , Represents the actual value; The L1 norm is used to provide feedback on the discrepancy between the predicted confidence plot and the actual value.

[0027] The aforementioned circuit pattern feature matching method based on dense matching, for key point detection, designs a key point loss function. In the formula, Indicates the predicted location of key points. Represents the actual location index; The larger the value, the greater the deviation in key point location detection. By improving key point detection, the value of the loss function can be lowered below the designed threshold.

[0028] In the aforementioned circuit pattern feature matching method based on dense matching, step S4-2 involves refining the fine-grained matching using the L2 loss function. In the formula, For tags, for The total variance of the key point heatmap was calculated. To achieve fine-grained matching results, and in order to eliminate quantization errors in keypoint localization and improve alignment accuracy, an L2 loss function based on uncertainty weighting is employed. Optimization is performed. This loss function not only calculates the predicted keypoints... With real labels The accuracy of the geometric position is constrained by the Euclidean distance between them, and key points are also introduced. Total variance of the heatmap corresponding to key points This serves as a dynamic weight to measure matching uncertainty. It is calculated by dividing by [a factor] during loss calculation. The model can adaptively adjust the gradient contribution of different samples: for low variance regions with clear texture and high confidence, it assigns greater optimization weights; while for high variance regions with blurred texture or ambiguity, it automatically attenuates their impact on the loss, thereby achieving sub-pixel-level accurate matching while suppressing noise interference.

[0029] Compared with the prior art, the advantages of this invention are:

[0030] (1) Potential key points in circuit graphics images are quickly determined by coarse-grained matching, and then the key points are finely adjusted by fine-grained matching to make the position of the key points more accurate, especially in high noise and low texture areas, which significantly improves the accuracy of feature matching.

[0031] (2) By generating a confidence map for each key point, and combining cross-correlation and confidence screening mechanisms, low-quality key points and mismatches are effectively removed, while high-confidence key points are retained, providing more reliable input data for subsequent defect detection.

[0032] (3) By aggregating features at different scales, using large-scale features to supplement the detailed information of small-scale features, and using small-scale features to provide smoother structural features, the complementarity of multi-scale features can effectively improve the utilization rate of features and further optimize the accuracy and stability of image reconstruction. Attached Figure Description

[0033] Figure 1 This is the overall system flowchart of the algorithm of the present invention, used for dense matching of circuit graphic features. Detailed Implementation

[0034] A circuit pattern feature matching method based on dense matching includes the following steps:

[0035] Step S1, Multi-scale feature fusion:

[0036] Image to be detected With template image Four multi-scale features were extracted using a CNN model. , , , , The resolutions are respectively the original image resolutions. , , , , , , These are the image's height, width, and number of channels, respectively. Representing the set of real numbers, ultimately in multi-scale features The fusion process is performed to obtain the low-resolution fused image to be detected. Template low-resolution fusion image , , , , These represent the height and width of the low-resolution fused image, respectively.

[0037] Step S2, Key Point Detection:

[0038] Multiscale features of two images First, each element undergoes a 1×1 convolution, increasing its channel dimension by 1. Then, it is flattened along the channel dimension to obtain a heatmap of the key points to be detected. Template key point heatmap , , .

[0039] Step S3: Extract the confidence map:

[0040] For the low-resolution fusion image to be detected obtained in step S1 Template low-resolution fusion image The confidence map of the target is obtained by passing through a linear layer. Template confidence plot , , This is used to evaluate which key points are effective.

[0041] Step S4, Feature Matching Stage:

[0042] The extracted key points need to be processed through two stages: coarse-grained matching and fine-grained matching.

[0043] Step S4-1, Coarse-grained matching stage:

[0044] First, combine the heat map of the key points to be detected. and the confidence graph of the target Template key point heatmap Template confidence plot Calculate the confidence score for each keypoint and select the top score. Keypoints with high confidence scores are identified, and interpolation methods are used to extract these keypoints to form feature descriptors. L2 normalization is then applied to the extracted feature descriptors, and finally, the normalized feature descriptors are adjusted according to the image scaling ratio to restore the resolution to the original image resolution. That is, to obtain the image descriptor to be detected. Template image descriptor ;

[0045] Calculate the image descriptor to be detected Template image descriptor The similarity between the two neighbors is used to perform nearest neighbor matching, and a rough matching result is obtained. , , It is the number of keypoints in each image descriptor. , These are the key points selected for matching between the two image descriptors.

[0046] Step S4-2, Fine-grained matching stage:

[0047] After obtaining preliminary results from coarse-grained matching, the process switches to processing the high-resolution original image. The high-resolution original image retains more detail, allowing for more accurate identification of key points and more refined matching.

[0048] Refined Alignment: In this stage, the preliminary matching results obtained from the coarse-grained matching stage are used to further refine the alignment of features. By using the positional information from the coarse matching as initial alignment guidance, the positions of corresponding features in the image can be adjusted more precisely, resulting in a more accurate match.

[0049] Furthermore, in step S1, for multi-scale features... By using a combination of upsampling and 1×1 convolution, the resolutions of the two multi-scale features are fused to the original image resolution. For multi-scale features Then, use DW convolution to reduce its resolution to half the original image resolution. Finally, the obtained multi-scale features are stitched together to obtain a low-resolution fused image. In order to maintain spatial resolution without sacrificing speed, the image to be detected is... With template image It is represented as a 2D grid consisting of 8×8 pixels on each grid cell, and each cell is reshaped into a 64-dimensional feature, preserving the spatial granularity within a single grid cell.

[0050] Furthermore, in step S2, multi-scale features Through processing by multiple convolutional or fully connected layers, a keypoint heatmap is generated to represent the locations of keypoints. This keypoint heatmap contains the probability or confidence level of each location within a grid cell as a keypoint. Its values ​​can be visualized by mapping the score of each pixel to color or grayscale values, thus forming a keypoint heatmap of the same size as the original image. In this keypoint heatmap, the intensity or color depth of a pixel reflects the probability of that location being a keypoint. High brightness indicates that the location has a high probability of being identified as a keypoint, while low brightness indicates that the location is less likely to be a keypoint.

[0051] Before generating the keypoint heatmap, the input multi-scale features After processing by the convolutional network, the number of channels is 64+1. The first 64 channels encode the location information of potential keypoints in the image, with each channel corresponding to a possible keypoint location in a grid cell. The additional 65th channel is designed as a dustbin channel to represent regions where no keypoints were detected. By adding a dustbin channel, the network is essentially providing a special mechanism to handle the case of no keypoints. By explicitly introducing this dustbin channel into the network, it can return a clear signal of no keypoints in regions where no valid keypoints are detected.

[0052] A 1×1 convolution is used to expand the number of channels, adding an extra channel for dustbin prediction. This convolutional layer is responsible for outputting 64+1 channels at the network's final layer, the same size as the input circuit image, with the 65th channel used to identify locations without keypoints. This design helps to accurately distinguish which regions lack keypoints during inference, avoiding misclassification of incorrect regions as keypoints. In this way, the network can not only accurately locate keypoints in the image but also identify regions without keypoints, thereby improving the model's robustness and ability to handle keypoint-free regions.

[0053] Furthermore, in step S3, the confidence map is generated to represent the confidence level of each location in the keypoint heatmap as a valid matching keypoint. The confidence map is derived from the low-resolution fused map, which is processed through a linear convolutional layer to obtain a confidence map with the same resolution and one channel as the low-resolution fused map. The confidence map is used for match filtering. During image matching and feature matching, the confidence map can be used to filter the quality of the matches: 1) Keypoints with high confidence are considered more reliable matches and can be retained for subsequent matching operations. 2) Keypoints with low confidence may be discarded to avoid incorrect matches and matching failures.

[0054] The role of confidence maps in feature matching is to improve the accuracy of image matching. During image matching, not only is the similarity of key points themselves considered, but the confidence information in the confidence map is also incorporated to determine which key points are more reliable in the matching process. This can effectively reduce the probability of false matches, especially in the presence of noisy or low-texture regions.

[0055] Furthermore, in step S4-1, coarse-grained matching is performed:

[0056] First, calculate the image descriptor to be detected. Template image descriptor The similarity matrix between them , The elements in the similarity matrix represent the similarity scores of each pair of keypoints. It refers to the batch size. It is the number of keypoints in each image descriptor. It is the dimension of the image descriptor. The dot product of formulas (1) and (2) yields the similarity of each pair of key points between two image descriptors.

[0057] (1)

[0058] (2)

[0059] In the formula, This represents an element in the similarity matrix of the image descriptors to be detected. This represents the elements in the similarity matrix of the template image descriptors. The maximum match between keypoints is selected from the similarity matrix, i.e., the most similar keypoints are found. Then, a mutual matching (MNN) strategy is performed, meaning a keypoint must be the most similar in both image descriptors.

[0060] (3)

[0061] (4)

[0062] (5)

[0063] In the formula, Representing dimension, , This indicates the key point matching result. This represents the matching results for the image descriptor to be detected. Key points and template image descriptor Key points ,if In template image descriptor The middle is the most similar, and Also in the image descriptor to be detected If the most similar key is found in the middle, then the match is valid. Finally, the indexes of the key points that satisfy the matching criteria will be returned. .

[0064] Furthermore, in step S4-2, fine-grained matching is performed:

[0065] Based on the results of coarse-grained matching, the matched keypoints on the low-resolution image descriptor are mapped back to the corresponding keypoints in the high-resolution original image. In the region, the image to be detected With template image corresponding The region is unfolded into a sequence of length 25, and the features obtained from two rounds of self-attention and cross-attention are then reassembled. Size feature map. The resulting image to be detected. of Feature map and template image of Perform point-by-point correlation operations on the feature maps to obtain their correlation maps, and then only focus on the image to be detected. of Central features of the feature map Get the corresponding template image of Which feature point in the feature map is used to calculate the degree of matching between image pairs through cross-correlation operations?

[0066] The relevance calculation results are normalized using Softmax, so that the similarity of each matching position can be represented as a probability. This allows us to select the most relevant matching point and assign a probability value to each candidate matching position, representing the confidence of its match. (Template image) of Selecting features with center in feature map Feature points with the highest degree of cross-correlation are matched with each other.

[0067] Furthermore, a series of related loss functions need to be designed to train the matching method. In step S1, for the two obtained low-resolution fused images to be detected... Template low-resolution fusion image Using Dual-softmax loss, (6), where, This is a similarity matrix. , Indicates the low-resolution fused image to be detected The Middle Key points and template low-resolution fusion map The Middle The key points are most similar.

[0068] For the confidence graph, a reliability loss was also designed, which represents the confidence level of local feature matching:

[0069] (7)

[0070] in, For the sigmoid function, For Hadamard products, , Represents the actual value.

[0071] A keypoint loss was designed for the keypoint detection part to supervise the keypoint detection branch, and the keypoint positions were mapped to a linear index. This indicates the location within the grid cell. If there are no keypoints in the grid cell, the dustbin category is used for labeling. The accuracy of keypoint detection is calculated using NLL loss:

[0072] (8)

[0073] In the formula, Indicates the predicted location of key points. This represents the actual location index.

[0074] In step S4-2, the L2 loss function is used for fine-grained matching to refine the level of detail. (9)

[0075] in, For tags, for The total variance of the corresponding key point heatmap was calculated. This is the result of fine-grained matching.

[0076] The present invention also provides a circuit pattern feature matching system based on dense matching, for implementing the circuit pattern feature matching method based on dense matching as described above, including a multi-scale feature extraction module, a confidence evaluation module, and a feature matching module; the multi-scale feature extraction module is used to extract multi-scale features of the input circuit pattern image, the confidence evaluation module is used to generate a confidence map and screen out reliable key points, and the feature matching module is used to perform coarse-grained matching and fine-grained matching to output high-precision matching results.

Claims

1. A circuit pattern feature matching method based on dense matching, characterized in that: Includes the following steps: Step S1, Multi-scale feature fusion: Image to be detected With template image Multi-scale features were extracted from each. , … ... Ultimately, in multi-scale features The fusion process is performed to obtain the low-resolution fused image to be detected. Template low-resolution fusion image For multi-scale features … Using a combination of upsampling and 1×1 convolution, multi-scale features are made available. … Resolution fusion into multi-scale features The resolution, for multi-scale features … Then, DW convolution is used to combine multi-scale features. … Resolution reduced to multi-scale features The resolution, and finally the multi-scale features , … … The images are stitched together to obtain a low-resolution fused image with fused features; Step S2, Key Point Detection: Image to be detected With template image Multiscale features Each element undergoes a convolution, and is then flattened along the channel dimension to obtain a heatmap of the key points to be detected. Template key point heatmap ; Step S3: Extract the confidence map: For the low-resolution fusion image to be detected obtained in step S1 Template low-resolution fusion image Each of these layers passes through a linear layer to obtain the confidence map of the target. Template confidence plot This is used to evaluate which key points are effective. Step S4, Feature Matching Stage: Step S4-1, Coarse-grained matching stage: Heat map of key points to be detected and the confidence graph of the target Combined template key point heatmap Template confidence plot By combining the confidence scores calculated for each keypoint separately, the top [keypoints] are selected. Keypoints with high confidence scores are identified, and interpolation methods are used to extract these keypoints to form feature descriptors. The extracted feature descriptors are then normalized, and finally, the normalized feature descriptors are adjusted according to the image scaling ratio to restore the resolution to multi-scale features. The resolution, i.e., the descriptor of the image to be detected. Template image descriptor ; Calculate the image descriptor to be detected Template image descriptor The similarity between them is used to perform nearest neighbor matching, and a rough key point matching pair is calculated. Step S4-2, Fine-grained matching stage: Fine-grained alignment: In this stage, the matching results obtained from the coarse-grained matching stage are used to further refine the alignment, more precisely adjusting the positions of corresponding key points in the image to obtain a more accurate match; based on the results of the coarse-grained matching, the matched key points on the low-resolution image descriptor are mapped back to the corresponding regions in the high-resolution original image, thus refining the image to be detected. With template image The corresponding regions are unfolded into sequences, and the features obtained through self-attention and cross-attention are then reassembled into a feature map; the image to be detected Feature maps and template images Perform point-by-point correlation operations on the feature map to obtain a correlation map, and then use the correlation map to extract the template image. Selecting from the feature map of the image to be detected Feature map center features Points with the highest cross-correlation are considered matched points, thus completing fine-grained matching; finally, the image to be detected is matched based on the results of fine-grained matching. With template image .

2. The circuit pattern feature matching method based on dense matching according to claim 1, characterized in that: In step S2, multi-scale features After convolution processing, the number of channels increases by 1. The extra channel is designed as a dustbin channel to represent regions where no keypoints were detected.

3. The circuit pattern feature matching method based on dense matching according to claim 2, characterized in that: In step S4-1, firstly, the image descriptor to be detected... Template image descriptor Performing a dot product yields a similarity matrix for the image descriptors. The elements in the similarity matrix represent the similarity scores for each pair of keypoints. , , The elements in the similarity matrix represent the image descriptors to be detected. Represents the elements in the similarity matrix of template image descriptors; The maximum match of image descriptors is selected from the similarity matrix, i.e., the most similar keypoints are found, and then a mutual matching strategy is executed. , This yields keypoint matching pairs ( ), and These are the image descriptors to be detected. Key points and template image descriptors in The key points in In template image descriptor Zhongyu It is the most similar, and In the image descriptor to be detected Zhongyu They are the most similar.

4. The circuit pattern feature matching method based on dense matching according to claim 3, characterized in that: For the obtained low-resolution fusion image to be detected Low-resolution fusion image with template Use the Dual-softmax loss function. In the formula, This is a similarity matrix. , Indicates the low-resolution fusion image to be detected The Middle Key points and template low-resolution fusion map The Middle The key points are most similar.

5. The circuit pattern feature matching method based on dense matching according to claim 4, characterized in that: For the confidence plot, design a reliability loss function. ,in, For the sigmoid function, For Hadamard products, , Represents the actual value.

6. A circuit pattern feature matching method based on dense matching according to claim 4 or 5, characterized in that: For keypoint detection, a keypoint loss function is designed. In the formula, Indicates the predicted location of key points. Represents the actual location index.

7. The circuit pattern feature matching method based on dense matching according to claim 6, characterized in that: In step S4-2, the L2 loss function is used for fine-grained matching. In the formula, For tags, for The total variance of the corresponding key point heatmap was calculated. This is the result of fine-grained matching.

Images