A semiconductor wafer surface defect multi-modal feature fusion visual detection method

Abstract

The present application relates to the technical field of visual detection, and particularly relates to a kind of semiconductor wafer surface defect multimodal feature fusion visual detection method, including with RT-DETR model as foundation, introduce MFM, CSN, MLA, construct RT-DETR-MCM model;With wafer defect data set RGB original wafer image as main mode, it is converted into gray scale chart, respectively extract Sobel gradient, Laplacian response and Canny edge, weighted fusion is structure chart, obtains structure mode;Main mode and structure mode are input into MFM module respectively, obtain cross-modal fusion feature;Cross-modal fusion feature is input into module CSN block, and adaptive enhancement for different scale feature response is realized;Then input MLA module, focus response to more reliable structure evidence area;The detection method of the present application starts from original wafer image, constructs derived structure mode, enhances the perception ability of model to edge mutation and local anomaly under the premise of not increasing additional sensor;The introduction of MFM, MLA, CSN module gives consideration to detection accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a visual inspection method for fusing multimodal features of defects on the surface of semiconductor wafers. Background Technology

[0002] As integrated circuit manufacturing processes continue to advance towards higher precision and higher integration, the surface quality of wafers has an increasingly direct impact on chip performance and yield. During manufacturing processes such as photolithography, etching, thin film deposition, and cleaning, wafers are prone to various surface defects, including particle contamination, scratches, cracks, ablation, and localized residues. Although these defects are usually limited in distribution, they often have a cumulative effect on subsequent processes, and in severe cases, can even lead to device failure. Therefore, timely and accurate identification of wafer surface defects during manufacturing is a crucial prerequisite for ensuring product quality and improving production efficiency.

[0003] Early wafer defect detection methods were mostly based on artificial feature design, typically using texture, grayscale, edge, or morphological information for defect identification. These methods are effective in specific scenarios, but they are sensitive to complex backgrounds, changes in defect morphology, and fluctuations in imaging conditions, and their generalization ability is limited.

[0004] In recent years, deep learning has demonstrated significant advantages in object detection tasks, driving the rapid development of industrial surface defect detection methods. One-stage detection models, represented by YOLO, excel in inference speed, while DETR-based methods, using ensemble prediction mechanisms, reduce reliance on anchor boxes and post-processing, exhibiting strong potential in frame uniformity and global modeling capabilities. In particular, RT-DETR (Real-Time Detection Transformer) achieves a good balance between end-to-end detection and real-time performance, providing a valuable technological foundation for industrial online inspection. However, most of these general-purpose detection models are designed for natural scenes, and when directly applied to wafer surface defect detection tasks, they are still affected by factors such as complex background textures, small target scale, and insignificant local features.

[0005] Compared with target detection in natural images, wafer surface defect detection has more prominent task specificity. (1) Many defect targets are small in size and account for a very low proportion in the image. After multi-layer downsampling, feature weakening is easy to occur, affecting the localization and recognition effect; (2) Wafer surfaces usually have large areas of repetitive and regular circuit textures. Such backgrounds may show high similarity to real defects in local areas, which can easily lead to false detection; (3) Some weak defects do not show obvious grayscale changes in the original image, and it is difficult to capture them stably by relying solely on appearance information.

[0006] The current solution to the above problems is to introduce additional imaging equipment and sensors to obtain multimodal features, but this also increases the size and computational load of the model, reduces the detection speed, and results in poor real-time performance. Summary of the Invention

[0007] The purpose of this invention is to provide a visual inspection method for multimodal feature fusion of semiconductor wafer surface defects, thereby addressing the problems existing in the prior art. To achieve the above objective, this invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a visual inspection method for multimodal feature fusion of semiconductor wafer surface defects, comprising the following steps:

[0009] S101. Based on the RT-DETR model, a cross-modal competitive fusion module MFM is introduced at the Backbone output, a cross-scale semantic aggregation module CSN is introduced at the front of the Neck, and a local adaptive enhancement module MLA is introduced at the middle of the Neck to construct the RT-DETR-MCM model.

[0010] S102. Using the original RGB wafer image in the wafer defect dataset as the main mode, convert it into a grayscale image, extract the Sobel gradient, Laplacian response and Canny edge respectively, and fuse them into a structure map to obtain the structural mode.

[0011] S103. Input the main mode and structural mode of the wafer defect image into the cross-modal competitive fusion module MFM to obtain cross-modal fusion features;

[0012] S104. Input the cross-modal fusion features into the cross-scale semantic aggregation module CSN to achieve adaptive enhancement of the feature response at different scales;

[0013] S105. Then input the Local Adaptive Evidence Enhancement (MLA) module to focus the response on the more reliable structural evidence region.

[0014] As a further technical solution, structural modalities are constructed to reveal edge anomalies and local geometric mutations hidden in the background in advance.

[0015] As a further technical solution, in step S103, the module MFM performs channel alignment and competitive weighting on the two representations in the multi-scale feature layer, suppresses redundant response caused by periodic textures and highlights complementary information, thereby obtaining cross-modal fusion features.

[0016] As a further technical solution, when the structural mode is more sensitive to the crack edge, the structural mode weight is higher; when the structural mode is disturbed by noise, the weight of the principal mode increases.

[0017] As a further technical solution, in step S104, the module CSN, based on cross-modal fusion features and combined with the upsampled semantic information of high-level features, achieves adaptive enhancement of feature responses at different scales through channel attention modeling, element-wise multiplication fusion, and residual summation operations.

[0018] As a further technical solution, in step S105, through module MLA, after local rearrangement, position competition and candidate weighting, the local responses of small defects are concentrated to improve the separability and recall of small targets.

[0019] In a second aspect, the present invention provides a controller, comprising:

[0020] A processor for implementing the detection method as described in the first aspect;

[0021] A communication interface used to communicate with other electronic devices.

[0022] Thirdly, the present invention provides a wafer defect detection system, comprising:

[0023] Image acquisition platform, used to acquire raw RGB images of the entire wafer;

[0024] The controller, as described in the second aspect, first generates a structural modality from the original RGB image, then takes the main modality and the structural modality as input and runs the method described in the first aspect.

[0025] Fourthly, the present invention provides a computer-readable storage medium storing computer program instructions that, when executed by a processing circuit, implement the method described in the first aspect.

[0026] Fifthly, the present invention provides a computer program product comprising instructions that, when the computer program product is run on a computer, cause the computer to perform the method as described in the first aspect.

[0027] The beneficial effects of the present invention are as follows:

[0028] (1) The detection method of this invention starts from the original wafer image and constructs a derived structural modality by fusing adaptive grayscale representation and structural responses such as Sobel, Laplacian, and Canny, thereby enhancing the model's ability to perceive edge abrupt changes and local anomalies without adding additional sensors. Based on this, a cross-modal competitive fusion module MFM is designed to suppress redundant responses caused by textures through channel-level soft selection; a local adaptive enhancement module MLA is introduced to strengthen more reliable structural evidence in a local range to improve the separability of small defects; at the same time, a cross-scale semantic aggregation module CSN is constructed to strengthen the information transmission between high-level semantics and shallow details and improve the detection stability in complex backgrounds.

[0029] (2) The detection method of this invention was tested on a publicly available wafer defect dataset. The results show that RT-DETR-MCM achieves mAP@0.5 and mAP@0.5:0.95 of 92.7% and 60.8%, respectively, and precision and recall of 91.5% and 90.2%, respectively. Its overall performance is superior to YOLOv7, YOLOv8, YOLOv10, YOLOv11 and the baseline RT-DETR. At the same time, the model computation is 22.7 G FLOPs, and the detection speed reaches 242.2 FPS, balancing detection accuracy and real-time performance. Furthermore, this method can effectively improve the identification ability of small and weak defects on the wafer surface, and has good industrial application value.

[0030] (3) The detection method of the present invention is different from the traditional multimodal method that relies on additional sensors to obtain multi-source data. The auxiliary mode (structural mode) used is derived from the original image. That is, the structural response map is constructed by adaptive gray-scale mapping and operators such as Sobel, Laplacian, and Canny, so that the model can obtain a supplementary representation that is more sensitive to edge and local geometric changes while learning the original appearance information.

[0031] (4) The MFM module introduced in this invention enables the main mode and structural mode to dynamically coordinate based on the feature response, rather than simply superimposing them. This can solve the problem of high redundancy and high false detection rate in multimodal fusion under complex circuit texture backgrounds. It uses competitive channel selection to make the fused features "cleaner" and more biased towards real defect evidence.

[0032] (5) The CSN module introduced in this invention is based on cross-modal fusion features and combines upsampled semantic information of high-level features. Through channel attention modeling, element-wise multiplication fusion, and residual summation operations, it achieves adaptive enhancement of feature responses at different scales. This design helps to strengthen the key texture and edge information of defective regions and improve the model's ability to detect wafer defects in small targets, complex textures, and strong reflective backgrounds.

[0033] (6) The MLA module introduced in this invention solves the problem that although defects exist, the local evidence is too weak and easily buried by the surrounding texture. It concentrates the local responses of small defects through local rearrangement, position competition and candidate weighting, thereby improving the separability and recall of small targets. Attached Figure Description

[0034] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof. It should also be understood that these drawings are for simplicity and clarity and are not necessarily drawn to scale. The invention will now be described and explained with additional features and details using the drawings, wherein:

[0035] Figure 1 A flowchart of the detection method in an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram of the MFM module structure in an embodiment of the present invention is shown;

[0037] Figure 3 A schematic diagram of the MLA module structure in an embodiment of the present invention is shown;

[0038] Figure 4 A schematic diagram of the CSN module structure in an embodiment of the present invention is shown;

[0039] Figure 5 A schematic diagram of the overall structure of the RT-DETR-MCM model in an embodiment of the present invention is shown;

[0040] Figure 6 The structural modal images in an embodiment of the present invention are shown. Detailed Implementation

[0041] The technical solutions in typical embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0042] Example 1

[0043] like Figure 1 As shown, this embodiment provides a visual inspection method for multimodal feature fusion of semiconductor wafer surface defects, including the following steps:

[0044] S101. Based on the RT-DETR model, a cross-modal competitive fusion module MFM (as shown in Table 1, the same below) is introduced at its backbone output, a cross-scale semantic aggregation module CSN is introduced at its neck front, and a local adaptive enhancement module MLA is introduced at its neck middle, thus constructing the RT-DETR-MCM model (MCM is an abbreviation for MFM, CSN, and MLA). Figure 5 As shown.

[0045] Table 1 Specific locations where modules are introduced

[0046] MFM Replace the P3, P4, and P5 single-channel outputs of the original single-modal ResNet18 backbone Backbone output / multimodal feature fusion input CSN Replace the direct upsampling + concat lateral fusion method of FPN in the original RT-DETR neck Neck front / top-down feature fusion stage MLA Replace the single-source feature inputs (P3, P4, P5) directly generated by the FPN and fed into the subsequent PAN aggregation and Decoder in the original RT-DETR. Neck mid-section / FPN-PAN input reconstruction stage

[0047] It should be noted that MFM (Multimodal Competitive Fusion Module), CSN (Cross-scale Semantic Aggregation Module), and MLA (Multimodal Local Adaptive Enhancement Module) are existing modules, such as... Figures 2-4 As shown, it will not be elaborated further.

[0048] S102. Using the original RGB wafer image from the wafer defect dataset as the main mode, convert it to a grayscale image (Equation 1). Extract the Sobel gradient (Equation 2), Laplacian response (Equation 3), and Canny edge (Equation 4) respectively, and weighted fuse them into a structure map to obtain the structure mode (Equation 5, also known as the second mode, such as...). Figure 6 (As shown).

[0049] (1)

[0050] (2)

[0051] (3)

[0052] (4)

[0053] The final structural mode can be written as:

[0054] (5)

[0055] In wafer surface defect detection, many defects are often small in scale, have low contrast, and are easily masked by large areas of repetitive textured backgrounds. While models can capture overall brightness and texture distribution by relying solely on the appearance information of the original RGB image, they are insufficient for depicting structural features such as cracks, breaks, grain edges, and local abrupt changes. Based on this characteristic, this embodiment further constructs derived structural modes on top of the main modal image. The aim is to extract more defect-sensitive geometric and boundary information from the original image, providing clearer structural priors for subsequent multimodal fusion.

[0056] The purpose of this step is not to perform direct detection, but to explicitly extract the prior structural information that is more sensitive to defects.

[0057] Sobel emphasizes the first-order gradient, which is more sensitive to linear, continuous abrupt changes such as scratches and cracks.

[0058] Laplacian emphasizes second-order changes and is more sensitive to localized sharp anomalies such as particles, pits, and burrs.

[0059] Canny extracts truly continuous fine edges, reducing texture interference in complex backgrounds.

[0060] In other words, this module addresses the problem of unclear defect structures in the original image. By constructing a second modality, it brings to the surface edge anomalies and local geometric abrupt changes that were originally hidden in the background, providing a cleaner and more defect-oriented input for the subsequent fusion module.

[0061] S103. Input the main mode and structural mode of the wafer defect image into the cross-modal competitive fusion module MFM respectively. Perform channel alignment and competitive weighting on the two representations in the multi-scale feature layer to suppress redundant response caused by periodic texture and highlight complementary information to obtain cross-modal fusion features.

[0062] In wafer defect detection, the background consists of high-density, highly periodic circuit layouts with textures exhibiting significant high-frequency details. When relying solely on a single appearance mode, small defects such as fine cracks, broken lines, and particles often closely resemble the background texture in their local responses, easily leading to situations where "defects are submerged by the texture" or "texture is misjudged as defects." Before multi-scale features are incorporated into subsequent local enhancement and cross-scale aggregation, a cleaner and more separable cross-modal representation is needed to improve the quality of subsequent token representations and query alignment stability. Introducing MFM (Multimodal Feature Fusion) allows for selective fusion of multimodal features through a "branch competition + channel-level soft selection" approach, avoiding redundant superposition and mode dominance issues caused by direct splicing / addition.

[0063] The MFM module does not perform direct fusion, but rather competitive fusion. It first aligns the channels of different modalities, then uses global pooling to obtain a fused summary, then uses a lightweight MLP to generate weights for each channel in two branches, and finally performs softmax on the branch dimension.

[0064] Its advantages are that when the structural mode is more sensitive to the crack edge, the structural mode weight is higher; when the structural mode is disturbed by noise, the principal mode weight will rise; for background textures that are strong in both paths, the softmax competition mechanism can avoid the "background doubling" caused by simple addition.

[0065] Therefore, the MFM module can solve the problem of high redundancy and false positives in multimodal fusion under complex circuit texture backgrounds. It uses competitive channel selection to make the fused features "cleaner" and more inclined to true defect evidence.

[0066] S104. Input the cross-modal fusion features obtained from the MFM module into the cross-scale semantic aggregation module CSN. Based on the cross-modal fusion features, combined with the upsampled semantic information of high-level features, adaptive enhancement of the response to features at different scales is achieved through channel attention modeling, element-wise multiplication fusion, and residual summation operations.

[0067] The CSN module is based on cross-modal fusion features from layers P3, P4, and P5, combined with upsampled semantic information from higher-level features. Through channel attention modeling, element-wise multiplication fusion, and residual summation, it achieves adaptive enhancement of feature responses at different scales. This design helps to strengthen key texture and edge information in defect regions, improving the model's ability to detect wafer defects in small targets, complex textures, and under strong reflective backgrounds.

[0068] In engineering implementation, the cross-layer feature integration module stacks and sums multi-path fused features to achieve initial convergence of cross-scale information. Multiply applies attention weights to the feature map through element-wise multiplication, thereby achieving explicit modulation in the channel dimension. Since the computational overhead of channel statistics and 1×1 transformation is relatively small, CSN can achieve cross-scale semantic fusion and channel selection without introducing costly global self-attention, making it more suitable for scenarios with high-frequency backgrounds and complex structures, such as wafer images. Furthermore, CSN can be effectively integrated with MFM and MLA to form a more coherent feature processing procedure.

[0069] The CSN module proposed in this embodiment is embedded in the early neck feature processing stage of multimodal RT-DETR. This module inherits the cross-modal fusion features output by MFM at the P3, P4, and P5 scales, and combines upsampled information from high-level semantic features. Through channel attention, element-wise multiplicative fusion, and residual summation mechanisms, it adaptively recalibrates the defect-related responses at different scales. The multi-scale features processed by CSN are further input into the MLA module for local adaptive enhancement with supplementary features generated from the original RGB / structural branches. This design can suppress periodic textures and reflective pseudo-edge interference before subsequent enhancement of FPN / PAN, improving the recall and localization stability of small target defects.

[0070] S105. Then, the Local Adaptive Evidence Enhancement (MLA) module is input. Through local candidate recombination and gated aggregation, the response is concentrated in a more reliable structural evidence region, thereby alleviating the problem of defect signals being submerged in strong background textures.

[0071] Even after MFM, where both modalities have undergone channel-level filtering, defects may still be very weak in the local space. Especially in areas with dense periodic textures, some defects may only occupy a small local area, and without finer-grained selection, the defect response will still be averaged out by the background.

[0072] This is a typical "local optimization enhancement" process.

[0073] It doesn't address the question of "which modality is more important," but rather "which sub-location is more like a real defect within a very small local block."

[0074] Therefore, it is particularly suitable for solving the following problems: tiny particles only fall on a local point; fine scratches only pass through a small local area; local weak defects have very low contrast and can only be seen by magnifying the local structural details.

[0075] Unlike regular attention, softmax competes within 2×2 sub-positions, meaning it doesn't raise the entire local block together, but instead prioritizes enhancing the position with the strongest evidence, thus reducing the risk of the texture background being enhanced as a whole.

[0076] Therefore, this module addresses the problem of defects existing but with weak local evidence that is easily obscured by surrounding textures. It improves the separability and recall of small targets by concentrating local responses to minor defects through local rearrangement, position competition, and candidate weighting.

[0077] Experimental Example 1

[0078] To verify the effectiveness of each key component (MFM, MLA, CSN) in the proposed multimodal RT-DETR-MCM architecture and their specific contributions to model performance, progressive quantitative and qualitative ablation experiments were conducted on a wafer defect dataset. The experiments used RT-DETR-ResNet18 as the baseline model and gradually introduced structural modality construction, the cross-modal competitive fusion module MFM, the local adaptive enhancement module MLA, and the cross-scale semantic aggregation module CSN.

[0079] Table 2 Ablation Experiment

[0080] MFM ✓ ✓ ✓ ✓ CSN ✓ ✓ ✓ ✓ MLA ✓ ✓ ✓ ✓ FLOPs (G) 57.2 23.3 48.1 76.0 20.5 25.8 69.4 22.7 Params (M) 19.89 15.33 18.20 33.89 16.15 21.80 32.45 20.27 FPS 227.27 166.6 212.76 88.49 248.60 185.30 98.50 242.22 mAP@50 (%) 88.7 89.8 89.3 90.9 90.4 91.7 91.2 92.7 <![CDATA[ mAP@0.5:0.95 (%)]]> 51.1 53.5 52.4 55.8 55.1 59.3 56.9 60.8

[0081] Based on the experimental data in Table 2, the following conclusions can be drawn:

[0082] Structural modes and the characteristic dimensionality reduction effect of MFM (Model I vs. Baseline):

[0083] After introducing the MFM module and structural modalities, the model's FLOPs significantly decreased from 57.2G to 23.3G, while mAP@50 improved by 1.1%. This result demonstrates that the channel competition mechanism in MFM effectively utilizes the sparsity of structural modalities, suppressing a large number of redundant periodic background texture features in the original RGB image. By filtering out invalid background information, MFM plays a role in feature selection and dimensionality reduction, thereby enhancing the discriminative power of features while reducing computational overhead.

[0084] The cost of local augmentation in MLA (Model III vs. Baseline):

[0085] While using the MLA module alone improved mAP@0.5:0.95 by 4.7%, it also caused FLOPs to surge to 76.0G and FPS to drop to 88.5. This confirms that directly applying local adaptive enhancement to unfiltered high-dimensional features incurs a huge computational burden, making it difficult to meet the requirements of real-time industrial detection.

[0086] Inter-module collaborative optimization mechanism (Model VI vs. Model V):

[0087] Comparing Model VI (without MFM) and Model V (with MFM) reveals a significant dependency between modules. In the absence of MFM (Model VI), MLA directly operates on high-dimensional features, resulting in a computational cost of 69.4 G FLOPs. However, with the introduction of MFM (Model V), despite the addition of a processing module, the overall FLOPs decrease to 25.8 G, and the FPS recovers to 185.3. This demonstrates that MFM, as a pre-module, successfully constructs a "feature funnel," ensuring that subsequent computationally intensive MLA only operates on selected low-dimensional effective features, thus resolving the high computational cost issue of MLA.

[0088] Final model performance balance (Ours):

[0089] After integrating the CSN module, the complete model achieves an optimal performance balance. CSN further optimizes the cross-scale feature flow, enabling the final model to achieve a peak inference speed of 242.2 FPS and an optimal detection accuracy of 92.7% while maintaining extremely low computational cost (22.7 G FLOPs). This demonstrates that the multimodal architecture proposed in this embodiment successfully achieves closed-loop optimization from Feature Purification (MFM), Local Detail Enhancement (MLA), to Global Semantic Convergence (CSN).

[0090] (a) Background interference suppression capability

[0091] When faced with highly periodic circuit textures, the baseline model exhibits diffuse attention, with a large number of activations concentrated on the normal background, causing defect features to be obscured. In contrast, the model in this embodiment demonstrates significant background suppression characteristics. Thanks to the explicit encoding of structural gradients by the MFM module, the model successfully ignores regular textures and accurately focuses high-response regions on the geometric edges of defects, verifying its robustness in complex backgrounds.

[0092] (b) Minimal defect detection capability

[0093] For low-contrast defects, the baseline model is prone to missed detections or location drift. However, after introducing the MLA module, the model in this embodiment forms a high-intensity activation hotspot in the core defect region, indicating that the local adaptive enhancement mechanism effectively amplifies weak feature signals and significantly improves the recall capability for small targets.

[0094] In summary, the ablation experiments fully verified the rationality and engineering applicability of the RT-DETR-MCM architecture design from both quantitative and qualitative visual dimensions.

[0095] Experimental Example 2

[0096] To evaluate the overall performance of RT-DETR-MCM in wafer surface defect detection, this embodiment conducts a rigorous comparative test with mainstream single-stage detection algorithms in the industry. The comparison models include YOLOv7, YOLOv8, YOLOv10, and YOLOv11 models, as well as the original RT-DETR model without the improved modules. To ensure fairness, all comparison models were run under the same hyperparameter configuration, hardware and software environment, and training strategy, and all used a dual-stream input method of "RGB + structural mode" to eliminate interference from differences in data sources.

[0097] Table 3 Comparative Experiments

[0098] RT-DETR-RGB&Str 87.6 86.5 88.7 51.1 48.2 33.89 92.6 Yolov7-RGB&Str 79.5 78.2 79.8 44.6 67.1 12.46 203.3 Yolov8-RGB&Str 85.1 81.5 86.1 51.3 13.3 5.33 225.0 Yolov10-RGB&Str 86.2 82.1 88.4 55.4 16.8 7.24 268.0 Yolov11-RGB&Str 84.5 83.4 84.6 50.2 12.3 5.13 305.0 RT-DETR-MCM (Ours) 91.5 90.2 92.7 60.8 22.7 20.27 242.2

[0099] Analysis of multiple metrics, including precision, recall, and mean precision (mAP@50 and mAP@0.5:0.95), leads to the following conclusions:

[0100] Referring to Table 3, reducing false negatives is crucial for ensuring the reliability of quality inspection in wafer surface defect detection tasks. As shown in Table 3, the recall rate of YOLO-based models (v7 to v11) did not exceed 84%. This is attributed to the inherent local receptive field limitation of convolutional neural networks, which makes it difficult to effectively decouple the background from weak defects when processing large-scale periodic textured backgrounds, resulting in high-frequency noise drowning out the defect signal during the feature extraction stage. In contrast, the RT-DETR baseline model, which introduces a global self-attention mechanism, improves the recall rate to 86.5%. The RT-DETR-MCM model proposed in this embodiment further improves this indicator to 90.2%. Compared with the baseline model and YOLOv10, the recall is improved by 3.7% and 8.1%, respectively. This performance gain indicates that the cross-modal competition mechanism of the MFM module effectively suppresses the feature response of background textures during the feature encoding stage, enhances the model's ability to extract subtle topological features such as scratches and broken lines, and thus significantly reduces the false negative rate.

[0101] mAP@0.5:0.95 is a comprehensive metric for evaluating the accuracy of bounding box regression in object detectors. Experimental data shows that YOLOv10, benefiting from its dual-label assignment strategy, achieves 55.4% on this metric, outperforming the original RT-DETR's 51.8%, indicating limitations of the baseline Transformer architecture in fine-grained localization of small targets. The RT-DETR-MCM model achieves a significant improvement on this metric, reaching 60.8%. This result confirms the effectiveness of the local adaptive enhancement strategy in the MLA module: by strengthening the edge gradient information of structural modes within a local window, the model enhances its ability to fit the geometric contours of defects, effectively addressing the problem of insufficient localization accuracy in bounding box regression tasks in the original DETR architecture.

[0102] Throughout the entire lifecycle, RT-DETR-MCM consistently ranks at the top of the performance envelope, being the only model to simultaneously break through 90% in both Precision and Recall metrics. This fully validates that the synergistic effect of MFM denoising, MLA enhancement, and CSN aggregation mechanisms has achieved a comprehensive leap in detection performance, rather than a trade-off of a single metric.

[0103] Example 2

[0104] This embodiment provides a controller, including:

[0105] A processor for implementing the detection method as described in Example 1;

[0106] A communication interface used to communicate with other electronic devices.

[0107] Example 3

[0108] Thirdly, this embodiment provides a wafer defect detection system, including:

[0109] Image acquisition platform, used to acquire raw RGB images of the entire wafer;

[0110] The controller described in Example 2 first generates a structural modality from the original RGB image, and then uses the main modality and structural modality as inputs to run the method described in Example 1.

[0111] Example 4

[0112] This embodiment provides a computer-readable storage medium storing computer program instructions, which, when executed by a processing circuit, implement the method described in Embodiment 1.

[0113] Example 5

[0114] This embodiment provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in Embodiment 1.

[0115] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A visual inspection method for multimodal feature fusion of semiconductor wafer surface defects, characterized in that, Includes the following steps: S101. Based on the RT-DETR model, a cross-modal competitive fusion module MFM is introduced at the Backbone output, a cross-scale semantic aggregation module CSN is introduced at the front of the Neck, and a local adaptive enhancement module MLA is introduced at the middle of the Neck to construct the RT-DETR-MCM model. S102. Using the original RGB wafer image in the wafer defect dataset as the main mode, convert it into a grayscale image, extract the Sobel gradient, Laplacian response and Canny edge respectively, and fuse them into a structure map to obtain the structural mode. S103. Input the main mode and structural mode of the wafer defect image into the cross-modal competitive fusion module MFM to obtain cross-modal fusion features; S104. Input the cross-modal fusion features into the cross-scale semantic aggregation module CSN to achieve adaptive enhancement of the feature response at different scales; S105. Then input the Local Adaptive Evidence Enhancement (MLA) module to focus the response on the more reliable structural evidence region.

2. The visual inspection method for multimodal feature fusion of semiconductor wafer surface defects as described in claim 1, characterized in that, By constructing structural modalities, edge anomalies and local geometric mutations hidden in the background can be made explicit in advance.

3. The visual inspection method for multimodal feature fusion of semiconductor wafer surface defects as described in claim 1, characterized in that, In step S103, the two representations are aligned and competitively weighted at the multi-scale feature layer by the module MFM, which suppresses redundant responses caused by periodic textures and highlights complementary information, thereby obtaining cross-modal fusion features.

4. The visual inspection method for multimodal feature fusion of semiconductor wafer surface defects as described in claim 3, characterized in that, When the structural mode is more sensitive to the crack edge, the structural mode weight is higher; when the structural mode is disturbed by noise, the weight of the principal mode increases.

5. The visual inspection method for multimodal feature fusion of semiconductor wafer surface defects as described in claim 1, characterized in that, In step S104, the CSN module, based on cross-modal fusion features and combined with upsampled semantic information of high-level features, achieves adaptive enhancement of feature responses at different scales through channel attention modeling, element-wise multiplication fusion, and residual summation operations.

6. The visual inspection method for multimodal feature fusion of semiconductor wafer surface defects as described in claim 1, characterized in that, In step S105, through module MLA, local responses of minor defects are gathered together after local rearrangement, position competition and candidate weighting, thereby improving the separability and recall of small targets.

7. A controller, characterized in that, include: A processor for implementing the detection method as described in any one of claims 1-6; A communication interface used to communicate with other electronic devices.

8. A wafer defect detection system, characterized in that, include: Image acquisition platform, used to acquire raw RGB images of the entire wafer; The controller of claim 7 first generates a structural modality from the original RGB image, then takes the main modality and the structural modality as inputs and runs the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processing circuit, implement the method as described in any one of claims 1-6.

10. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-6.

Images