Small target defect recognition model training method based on deep learning

Abstract

The present invention discloses a method for training a small target defect recognition model based on deep learning, which relates to the technical field of defect recognition model training. Focusing on the needs of small defect detection, it starts with the construction and precise labeling of high-resolution diversified data, highlights small targets through multi-scale feature fusion and spatial attention, and handles difficult scenarios with layer-by-layer screening and secondary reinforcement training. It also adopts multi-model fusion and online dynamic tuning to achieve real-time iterative optimization, and finally expands to multi-modal and time series dimensions to capture deeper and dynamic defect information, greatly reducing the missed detection and false detection rate, and improving the detection efficiency and adaptability of micron-level defects in complex process backgrounds; and further with the help of multi-source data such as infrared, X-ray or 3D morphology and time series modeling methods, and integrates multiple dimensions and incorporates hidden or early cracks into the detection and prediction range, thereby constructing a highly reliable and evolvable small target defect intelligent recognition system.

Description

Technical Field

[0001] The present invention relates to the technical field of defect recognition model training, and specifically to a small target defect recognition model training method based on deep learning. Background Art

[0002] In modern semiconductor manufacturing, as integrated circuit feature sizes continue to shrink and circuit structures become increasingly complex, the types and distribution patterns of defects appearing on wafer surfaces are becoming increasingly diverse. In high-end processes, not only can conventional defects such as metal particle adhesion, microcracks, and etching residues occur, but they are also often accompanied by microvoids or extremely subtle structural anomalies in critical locations with negligible areas. These defects are typically distributed within complex wiring metal layers, dielectric layers, or via areas. Actual wafer production lines often face the simultaneous demands of high-volume, high-resolution online inspection, requiring the identification and screening of even the smallest possible defects within a short period of time. Failure to promptly identify and remove wafers with potential risks will not only impact the reliability of subsequent process steps (such as chemical mechanical polishing, interlayer dielectric deposition, and packaging bonding), but can also directly lead to potential failure of the entire wafer or downstream devices. Correspondingly, when the process node is upgraded to a smaller line width, the visual inspection system's capture accuracy and sensitivity for micron-level and submicron-level defects also need to be improved simultaneously. Otherwise, it will face extremely high risks of missed detection and misjudgment costs, making it difficult to meet the demands of high-yield and high-capacity modern wafer manufacturing.

[0003] A search revealed a Chinese invention patent application with publication number CN115775236A, which discloses a method and system for visual detection of surface micro-defects based on multi-scale feature fusion. The method includes the following steps: acquiring an image set of the target object to be detected, and constructing an image training dataset for industrial product micro-defects based on the image set of the target object to be detected; training a multi-scale feature fusion deep neural network based on the image training dataset to obtain an optimized deep neural network; and inputting the target object image into the optimized deep neural network for processing to obtain an image of the target object defect. By adopting an attention mechanism and a multi-scale feature fusion network structure, this application can effectively suppress information loss caused by downsampling and improve the recognition rate of industrial product micro-defect detection. Furthermore, by adding a small target detection layer, micro-defects in industrial products can be more accurately detected.

[0004] Currently, the industry widely uses automated optical inspection (AOI) equipment combined with machine vision algorithms for rapid positioning and screening of chip wafers for in-line inspection. However, for micron-level or even smaller targets, when placed within circuit patterns with complex textures and noise components, deep learning models often fail to effectively identify defects due to insufficient training samples or feature scale mismatch, resulting in a significant increase in missed detections.

[0005] Especially in high-density wiring areas or critical structural levels, any microcracks or irregular holes left unnoticed can further expand in subsequent steps, ultimately leading to product failure and causing serious economic losses and quality risks. Relying solely on increasing the detection threshold or enhancing image magnification will not only easily introduce more false alarms and reduce production line efficiency, but will also significantly increase data processing volume and reduce overall inspection speed. Therefore, how to improve the reliability of detecting extremely small defects in multiple dimensions from image acquisition, model training, and online discrimination has become a core demand of the current advanced semiconductor manufacturing industry and a technical bottleneck that needs to be overcome urgently. Summary of the Invention

[0006] (1) Technical problems solved

[0007] In response to the shortcomings of the existing technology, the present invention provides a small target defect recognition model training method based on deep learning, which highlights small targets through multi-scale feature fusion and spatial attention, and handles difficult scenarios with layer-by-layer screening and secondary reinforcement training. It also uses multi-model fusion and online dynamic tuning to achieve real-time iterative optimization, and finally expands to multi-modal and time series dimensions to capture deeper and dynamic defect information, greatly reducing the missed detection and false detection rate, and improving the detection efficiency and adaptability of micron-level defects in complex process backgrounds; and further with the help of multi-source data such as infrared, X-ray or 3D morphology and time series modeling methods, and integrates multiple dimensions and includes hidden or early cracks into the detection and prediction range; thereby solving the technical problems recorded in the background technology.

[0008] (2) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a deep learning-based method for training a small target defect recognition model, including high-resolution sampling and pixel-level annotation of real and synthetic image sets, and generating multi-channel label mapping through unified ID management and high-resolution preprocessing to construct initial data that encompasses the diversity of rare small defects;

[0010] Multi-scale feature fusion and spatial attention mechanisms are introduced to the image features in the storage container to significantly amplify fine-grained targets at the fused feature level and ensure that multi-layer semantics and edge details form a learnable context;

[0011] Based on the preliminary detection model, the image is inferred. If a candidate defect area with low confidence or high background interference appears, the image corresponding to the candidate list is resampled and stored in the difficult example set. The refined sub-model is generated by combining reinforcement learning factor training;

[0012] A fusion strategy is used to cross-check IoU voting, comprehensive confidence evaluation, and background interference. Online dynamic tuning and secondary scanning are performed on low-confidence areas, providing real-time feedback on new difficult examples and enabling continuous self-learning.

[0013] After acquiring multimodal information or time-series images to track defect evolution, cross-channel attention and time-series features are added to the fusion feature and refinement sub-model to fuse multi-source elements to capture deep defect characteristics and provide early warning of crack trends.

[0014] Furthermore, high-resolution images are acquired from the wafer and labeled. Generative adversarial networks are then used to embed defect templates into actual wafer backgrounds or random textures to obtain synthetic data.

[0015] All real and synthetic data are saved in a data container, and the defect coverage calculated in real time is written into the metadata file together with the original resolution information of each image.

[0016] Furthermore, after two rounds of independent annotation for the same defect, the annotation results are checked for consistency, and the consistency score is used to measure the overlap between different annotation rounds;

[0017] If the consistency score is greater than the preset consistency threshold, the annotation is accepted; otherwise, the defect is included in the abnormal annotation list, triggering the re-annotation or expert review process, which will obtain the defect information and update the storage container.

[0018] Furthermore, the labeled images and defect information are input into the basic backbone of the convolutional neural network to obtain shallow features and deep features; a layer fusion module is introduced to upsample or downsample and align the above two, and a fused feature map is obtained after splicing or weighted summation. During the fusion, a multi-scale weighting function can be defined to perform nonlinear adjustment on the features of each scale.

[0019] Furthermore, the fused feature map is input into the spatial attention module to learn trainable attention maps for different regions in the image, and is multiplied pixel by pixel with the fused feature map to generate spatial attention enhanced features;

[0020] During the training phase, the internal parameters of the global memory module and activation function are continuously corrected. When the attention module, detection head and backbone network are trained together, the preliminary detection model is obtained.

[0021] Furthermore, large-scale wafer surface images are input for inference, and the preliminary detection model outputs a confidence value for each candidate defect; low-confidence defects are screened out using a global judgment threshold;

[0022] The background confusion degree is used to measure the interference strength between the fused features in the candidate defect area. When the background confusion degree exceeds the preset confusion threshold, the corresponding candidate defect area is marked as a background easily confused area; the output is a candidate list consisting of overlapping low-confidence defects and background easily confused areas.

[0023] Furthermore, high-resolution resampling or local high-magnification scanning is performed on the image areas marked as easily confused by the background or low-confidence defects in the candidate list to obtain clearer sub-images and store them in the difficult image pool;

[0024] The difficult example images are annotated again at the pixel level and an annotation file is generated. A special data enhancement strategy for extremely small objects is adopted, and a reinforcement learning factor is introduced to perform secondary training on the refined sub-model.

[0025] Furthermore, the detection frames of the preliminary detection model and the refined sub-model are subjected to IoU voting and confidence synthesis through a fusion function to obtain a global detection result, so as to screen out frames with too low confidence or high overlap, and use the result for real-time difficult identification and secondary scanning.

[0026] Furthermore, the output with a confidence level lower than the confidence threshold is determined as a difficult sample and added to the real-time to-be-scanned list;

[0027] Perform secondary acquisition or high-magnification imaging on the defect areas in the real-time to-be-scanned list; the obtained secondary scan images are input into the fusion inspection process again, and the updated inspection results overwrite the original suspected results;

[0028] If there are still defective areas that the system cannot determine after the second scan, they will be recorded in the online difficult example database and a conservative strategy will be adopted temporarily.

[0029] Furthermore, the online difficult example database and the final confirmed labels by manual or high-magnification microscopy are incorporated into the training pipeline, and the weight function is used to incrementally update the refined sub-model or part of the network weights of the two models; based on the model performance after online incremental learning, the fusion strategy and confidence threshold are automatically or semi-automatically updated.

[0030] Furthermore, the newly added multimodal imaging data and 3D morphology measurement results are stored in the expanded data container through spatial calibration and time stamp alignment; the coordinates of the infrared or X-ray image are aligned with the optical image using a multimodal mapping function, and the grayscale or pseudo-color method is unified.

[0031] Furthermore, an additional feature extraction branch is constructed for each new modality to form a multi-channel input, which is processed through cross-modal fusion mapping: after weighting the different modal information of the cross-channel attention module, the fusion obtains the multimodal cross-channel attention enhanced feature.

[0032] Furthermore, after the multimodal feature output, a temporal convolution module is added to integrate and predict the defect features of each frame in the time domain, and the defect evolution trend score is calculated through the temporal feature aggregation function; if the obtained trend score exceeds the corresponding alarm threshold, a higher-level review and maintenance process is automatically triggered.

[0033] (3) Beneficial effects

[0034] The present invention provides a deep learning-based small target defect recognition model training method, which has the following beneficial effects:

[0035] By building a data container in the first step The high-resolution and diverse defect data resources, supplemented by precise annotation, not only fundamentally alleviate the problem of scarcity of small target defect samples, but also provide a unified coordinate and identification system for subsequent steps;

[0036] In the second step, multi-scale feature fusion and spatial attention coordination mechanism are introduced to achieve the multi-scale weighted function. and spatial attention module As the core, we strengthen the interaction between shallow details and deep semantics, so that tiny targets can be is fully highlighted;

[0037] The third step is to perform secondary sampling and labeling of low confidence and easily confused areas on top of the above basic network, and combine it with reinforcement learning factors , forming a specialized refined sub-model , targeted reinforcement of difficult defects, and preliminary detection model Complementary enhancement of adaptability to extreme scenarios;

[0038] The fourth step is to use the fusion strategy The prediction results of the two models are integrated in real time during the detection phase, and online dynamic tuning is used to perform secondary scanning and incremental training of difficult samples, ultimately enabling the sustainable evolution of the detection system. This closed-loop system not only achieves high accuracy but also effectively reduces missed detection and false detection rates.

[0039] The fifth step further integrates multimodal information such as infrared, X-ray or 3D morphology, as well as dynamic modeling ideas such as temporal convolution and LSTM, into data containers and cross-channel attention. In the structure, the internal hidden cracks or early crack evolution can be visualized and predicted;

[0040] This overcomes the shortcomings of single-modality and single-timeframe detection in perceiving deep or dynamic defects, and fully incorporates new defect patterns through reinforcement learning and online dynamic tuning, forming a data-model-self-learning trinity. As a result, this system offers greater sensitivity, enhanced reliability, and significant scalability when processing extremely small defects in complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the small target defect recognition model training method of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1 The present invention provides a small target defect recognition model training method based on deep learning, including:

[0044] Step 1: High-resolution sampling and pixel-level expert annotation of real and synthetic image sets. Through unified ID management and high-resolution preprocessing, multi-channel label mapping is generated to build an initial data foundation that can fully capture the diversity of rare and subtle defects.

[0045] The step 1 includes the following:

[0046] Step 101: Build a high-resolution and diverse defect data source:

[0047] Generates defect data from two dimensions: real production environments and synthetic data, striving to improve both coverage and resolution. This is achieved through the interaction of the following technical features:

[0048] Acquire high-resolution images from wafers with different process nodes, circuit structures, and backside processing methods;

[0049] Ensure that the magnification meets the requirements of capturing submicron features during acquisition, and set a unique identifier for each image ,in is the increasing sequence number of the image in the real data source;

[0050] Using generative adversarial networks (GANs) or real defect cropping and splicing techniques, defect templates are embedded into actual wafer backgrounds or random textures to expand the diversity of defect distribution and background types. Real defect cropping and splicing involves cropping actual defect areas (including microcracks, metal particles, and holes) from real production line images and splicing these defect fragments onto different, defect-free wafer background images to generate new training samples.

[0051] In order to measure the distribution and fusion of different types of defects in diverse backgrounds in synthetic data, the defect coverage is defined as for:

[0052]

[0053] Where: Represents the normalized or mapped background-image continuous domain. Usually, the real / synthetic image space can be discretized into a two-dimensional or three-dimensional grid, and then mapped to middle;

[0054] express The coordinate vector of any point in the continuous domain is as follows: or wait;

[0055] Represents the total number of defect types considered in the synthetic data, such as metal particles, microcracks, pores, etc.; corresponding parameters are set for each type of defect and weight , ensuring that the evaluation function can comprehensively reflect the coverage and visibility of all defect types in the background; is a set of changing parameters of this type of defect in different sizes and shapes;

[0056] For the The weighting coefficient of the defect class in the overall synthesis difficulty or detection difficulty; Indicates the The background texture set corresponding to the class defect; is the defect category complexity index adjustment factor, and its value is greater than 0;

[0057] is the defect coverage mapping function, which can be implemented through matrix operations or vector inner products, etc., and is used to measure the Class defects in the background In the middle, by the set of changing parameters Under the action of the defined deformation / size parameters, the position The degree of coverage or distinguishability brought about; can be regarded as , for different locations Returns a non-negative real number. For example: when the defect is at this position When there is a strong occupancy or salience, The value is larger; if the defect is similar to the background or there is no defect at that location, then The value tends to 0.

[0058] By calculating and dynamically adjusting defect coverage , it is possible to specifically increase or reduce certain defect-background combinations, thereby finely controlling the balance between diversity and authenticity of the synthetic data, enabling subsequent training to learn more discriminative feature expressions in a richer sample environment;

[0059] All real data and synthetic data are saved in a unified high-resolution data container and record its unique identifier for each image or synthetic logo ,in is the sequence number of the composite image;

[0060] The defect coverage calculated in real time Write the metadata file together with the original resolution information of each image; introduce defect coverage between real data and synthetic data at the same time The unique identification system not only ensures data diversity, but also enables all images to be seamlessly associated and traced in subsequent processing (such as high-precision annotation and model training). Through the large-scale integration of multi-process real scenes and synthetic defects, combined with defect coverage Dynamic adjustment can effectively increase the diversity of defects while ensuring the controllability of image texture and defect morphology, and overall expand the perception space of defect detection for new modes.

[0061] Step 102: Multi-scale accurate labeling and labeling information output:

[0062] The data container generated in step 101 On this basis, the focus of step 102 is to achieve the refinement and traceability of the annotation through high-precision means, ensuring that the subsequent training phase can accurately learn the true tiny defect locations and boundaries, as follows:

[0063] For each original or synthesized image in step 101, high-magnification microscopy (or other micro-nanoscale imaging technology) is first used for local verification. When a suspected defect is identified, the surrounding area is rechecked at high magnification.

[0064] Based on the obtained review results, the defect location and shape are marked with pixel coordinates as the granularity, and the corresponding marking file is generated , and mapped with the unique ID of the image, for example: or This ensures that there are no omissions or mislabeling when fine-grained marking of extremely small defects, and the marking information is saved one-to-one. Subsequent steps do not require additional coordinate or boundary matching operations when calling.

[0065] Based on reference images provided by professional inspection instruments, or through a cross-validation mechanism involving multiple annotation experts, consistency checks are performed on previous high-magnification microscopic annotation results. If any defect annotations are found to deviate significantly from the actual boundaries, they are added to the abnormal annotation list and locally recalibrated until all experts reach a consensus on the defect location and boundaries.

[0066] In order to improve the quantitative level of small object annotation quality, consistency scoring is introduced , calculate the overlap between different annotation rounds (between experts or between instruments):

[0067]

[0068] Where: Indicates the number of defects verified; and Respectively represent The defects are in the binary masks of manual annotation and reference annotation; For the The weight of each defect in terms of its smallness or complexity, ranging from 0 to 1;

[0069] Used to measure the detail fit between two masks (can be based on a combination of multiple features such as edge contour superposition rate, center offset distance, etc., but does not use simple variance or standard deviation forms), and measure annotation differences in a higher dimension;

[0070] is the coupling coefficient of the multi-annotation mutual information, which is greater than 0 and is used to amplify the differences in small areas;

[0071] If the consistency score Greater than the preset consistency threshold , then the annotation results of the defect are considered to be consistent and the annotation is accepted; otherwise, the defect is included in the abnormal annotation list and the re-annotation or expert review process is triggered;

[0072] With the help of consistency scoring The measurement and cross-validation strategy can ensure that even the smallest defects can be repeatedly checked, significantly reducing errors such as annotation boundary drift or defect type confusion, and making the data annotations used in subsequent model training more consistent.

[0073] Finally, the defect information confirmed by the above annotation methods will be generated into a complete annotation metadata file , and store it in a unified data container in a one-to-one correspondence with the image, update and record it as the storage container ;

[0074] In the annotation metadata file For each defect, save its pixel-level coordinates, morphological label, and defect coverage from step 101. Key parameters such as , so that subsequent steps can directly call and utilize these marking information.

[0075] Consistency score The explicit quantification of the multi-annotation fusion process not only ensures the accuracy of the defect boundary, but also allows visualization and statistical management of the annotation process, which is a high-precision approach rarely used in traditional target detection data annotation. After the labeling is completed, it contains pixel-level information of extremely small defects and complete historical traceability information (all operation records from step 101 to step 102), providing traceable and high-precision data support for subsequent model training and online tuning. and information overlap A two-way interactive mechanism is formed in the data construction and annotation process: if it is found that the defect coverage is insufficient or the multi-annotation conflict rate is abnormally increased during annotation, it can be traced back to step 101 to fine-tune the configuration of the synthetic data so that the final output storage container Achieve a dynamic balance between defect diversity and annotation accuracy.

[0076] Step 2: When it is necessary to extract local details of extremely small defects and global contextual semantics in parallel, a multi-scale feature fusion and spatial attention module is introduced to the image features in the storage container. Other processing methods to integrate features It significantly amplifies fine-grained targets and ensures that multi-layer semantics and edge details form a learnable context;

[0077] The second step includes the following:

[0078] Step 201: Multi-scale feature extraction and fusion:

[0079] The storage container output from the first step Obtain the labeled image and defect information from the image and input it into the basic backbone of the convolutional neural network (such as 、 In this process, the features extracted from the initial layers are recorded as shallow features. , corresponding to the low-level texture and detail edges of the image; the features extracted by the subsequent deeper network are recorded as deep features , mainly including global semantics and contextual information. The reason for splitting shallow and deep features is to avoid the traditional detection network focusing only on high-level semantics and ignoring the morphological features brought by detailed boundaries when detecting small objects.

[0080] Introduce a cross-layer fusion module similar to FPN (Feature Pyramid Network) or Bifusion structure to combine feature maps at different resolutions (such as shallow feature maps) and deep features ) are upsampled or downsampled and aligned to the same size, and then concatenated or weighted summed in the channel dimension to obtain a fused feature map ;

[0081] Multi-scale weighting functions can be defined during fusion By performing nonlinear adjustments to the features of each scale, this function can be formalized as:

[0082]

[0083] Where: Indicates the number of layers that need to be fused in the network; For the Feature map of layer output; For use according to Adaptively weight the layer features based on their saliency or resolution;

[0084] Indicates the nonlinear mapping after overall fusion (for example Convolution or more complex transformations) to avoid information conflict caused by direct linear superposition;

[0085] In traditional methods, simple weighting or direct addition is often used to complete multi-layer feature fusion, which can easily cause shallow noise to be amplified indiscriminately or deep semantics to lose edge details. This solution uses By assigning different nonlinear adjustment factors to each layer, the network can flexibly adapt to the feature contribution of different depths, for example:

[0086]

[0087] Where: It is a nonlinear activation function, such as SiLU and GELU;

[0088] Trainable convolution kernels or linear mappings are used to extract key channels of features at each layer;

[0089] According to the level number An exponential amplification or reduction factor applied, the hierarchical exponential adjustment factor Can be optimized during training;

[0090] By combining trainable mapping with a hierarchical exponential amplification mechanism, shallow local edge information that is most suitable for identifying tiny defects and deep contextual information can be more accurately integrated.

[0091] When used, at the multi-scale fusion level, it effectively alleviates the problem of traditional detection networks ignoring tiny target features, allowing shallow texture information to receive sufficient attention and retention in weak backgrounds. In terms of network representation capabilities, the nonlinear adjustment factor combined with the exponential strategy significantly increases the flexible control of features at each layer, and can specifically amplify key channels suitable for detecting micro-defects.

[0092] When introducing spatial attention, the fusion features It will be used as the main input feature. If the quality of the fused feature itself is high enough, the attention mechanism can perform more detailed enhancement on the tiny defect area, and the overall effect will far exceed the sum of the shallow features or deep features used alone.

[0093] Step 202: Spatial Attention Coordination Mechanism:

[0094] The output fusion feature map Input spatial attention module ;Spatial attention module Trainable attention maps are learned for different regions of the image, including potential locations of small defects ; Size of attention map and fusion features Consistent, used to assign adaptive weights to each pixel / position;

[0095] Therefore, the updated fusion feature map It can be defined as:

[0096]

[0097] Where: is the spatial coordinate vector of the fusion feature; Represents the above fusion feature map (or fused feature map after attention enhancement )’s domain or spatial index set;

[0098] Represents element-wise multiplication; It is intended to ensure that the attention value is positive and adjusted around 1. If a pixel area is important for detection, then Take positive values ​​and amplify, otherwise it will weaken;

[0099] Through such adaptive operations, the area where the tiny defects are located can be highlighted in the feature map, while the response of the complex background or irrelevant area can be suppressed. The conventional spatial attention module only uses convolution or fully connected layers for rough generation. This solution obtains the attention map by introducing a combination of local attention and global memory. , ensuring a stronger ability to capture tiny defects:

[0100]

[0101] in: Represents the extraction of local fine-grained features;

[0102] It is an optional global memory module used to store the typical defect morphology or texture priors constructed in the first step, so as to quickly generate a high attention response when similar features appear;

[0103] Represents channel-level fusion operations, such as concatenation or summation;

[0104] To finally generate the activation function of the attention map, flexible nonlinear mappings can be used (such as Sigmoid mapping, Softmax spatial normalization, and hybrid Sigmoid+Softmax);

[0105] During the training phase, by comparing with the labeled data obtained in the first step, the model (the attention generation subnetwork injected between the backbone feature extraction network and the detection head, which also contains a global memory module) The memory matrix and activation function The convolution / MLP parameters) will continuously modify the global memory module and activation function The internal parameters of can improve the attention to various minor defects; when the attention module and the detection head are trained together with the backbone network based on the labeled data constructed in the first step, the overall network weights obtained constitute the deep network model.

[0106] The final fusion feature with spatial attention enhancement , which will be used for the subsequent third step of layer-by-layer screening and secondary reinforcement training. At this point, the defect information at all positions has been amplified at the feature level, and the background noise has been suppressed to a certain extent. In order to match the third step of re-judging the background easily confused area, the attention map is retained in the output and fusion features The corresponding spatial coordinate index, so that in the third step when making confidence judgments, the attention value can be directly called to determine whether an area is worth scanning again.

[0107] When used, local convolution features are combined with global memory to allow rare or extremely small defects to be amplified in the spatial attention map, which can more accurately highlight extremely small defects; non-defective areas are effectively suppressed, thereby reducing false detections, improving the reliability of the detection model for noisy backgrounds, and enhancing the ability to distinguish complex backgrounds; if there is only multi-scale fusion or only spatial attention, it is impossible to simultaneously take into account the details of small targets and the effective filtering of background noise; this solution, through the combination of steps 201 and 202, with the support of existing high-resolution data, realizes multi-dimensional regulation and spatial adaptive attention of multi-layer features, further amplifying the detection capability of small defects.

[0108] Step 3: Based on the preliminary detection model When a candidate defect area with low confidence or too high interference with the background appears after reasoning on the image, it will be removed from the candidate list. The corresponding images are resampled and stored in a subset of difficult examples, and then combined with reinforcement learning factor training to form a refined sub-model And significantly reduce the probability of missed detection and false detection of small targets.

[0109] The step three includes the following:

[0110] Step 301: Preliminary reasoning and candidate defect screening:

[0111] The deep network model output from the second step is recorded as the preliminary detection model , input large-scale wafer surface images for inference. The model has high sensitivity under the collaborative mechanism of multi-scale feature fusion and spatial attention, and can quickly detect areas with a high probability of defects;

[0112] During the inference process, the model is initially tested A confidence value is output for each candidate defect ; This step is determined by the global threshold Filter out suspicious areas, where: , the corresponding candidate area is regarded as a low-confidence defect;

[0113] Since tiny defects are often similar to background textures, when the confidence Close to the global decision threshold When the defect area is large, it is more likely to cause misjudgment or missed judgment; Fusion features The background complexity and interference intensity between them increase the background confusion of the background interference intensity ,in Indicates the The spatial position and morphological information of each candidate defect in the original image coordinate system:

[0114]

[0115] in: Indicates candidate defect areas In the mapping area of ​​the feature map space, it can be seen as the fusion feature A continuous or approximately continuous sub-region of pixels adjacent to the defect; express The coordinate vector of any position in or etc.), corresponding to the fusion features Pixels or feature units in ;

[0116] is the local feature covariance matrix, used to characterize The texture complexity near The covariance matrix of , to quantify the correlation and variation between the local textures;

[0117] Represents the Frobenius norm, which can be used to calculate the local feature covariance matrix Take the sum of all the squares of the elements and then take the square root; is a nonlinear exponential factor, and its value is greater than 0;

[0118] is an exponential mapping function used to convert attention saliency (Fusion feature map In spatial position The characteristic value of is introduced into the background interference determination link:

[0119] A trainable or designable mapping function, such as a small fully connected (MLP) mapping, is based on The characteristic response at the location determines the additional amplification / suppression;

[0120] When the background confusion Exceeding the preset confusion threshold When , the candidate defect area is marked as easily confused with the background;

[0121] This step outputs a candidate list consisting of overlapping low-confidence defects and background confusable areas. The list contains the defect coordinates and confidence levels that need to be focused on in subsequent reinforcement learning. , background confusion and other information.

[0122] When in use, the areas with low confidence and high background interference are managed uniformly to ensure that the most easily confused situations for tiny defects are not missed, and the defect area can be accurately located and determined; and defective areas The matching can realize the joint characterization of attention distribution and background complexity, and quickly focus on the tiny defective parts that may be wrong in the real reasoning stage. It will become the core basis for constructing a subset of difficult examples, reflecting the strong connectivity and traceability across steps.

[0123] Step 302: Construction of a subset of difficult examples and training of refined sub-models:

[0124] Based on the candidate list generated in step 301 , for the candidate list Perform high-resolution resampling or local high-magnification scanning on image areas marked as easily confused by background or low-confidence defects to obtain clearer sub-images and store them in the hard-to-image pool ;

[0125] In the resampling process, in order to take into account multi-scale information, multiple images can be collected at different magnifications for the same area (e.g. 、 、 ), and record the corresponding magnification parameters , used to distinguish scale differences in subsequent feature extraction and network training;

[0126] These difficult example images are re-calibrated at the pixel level or reviewed by experts to ensure that they are in the difficult example image pool. The accuracy of defect information in the annotation is ensured, and all annotation results are saved in the corresponding annotation files. , and the original image or Maintain the mapping relationship to avoid data confusion during subsequent merge training.

[0127] After obtaining a pool of hard-example images and its annotation files Finally, a special data enhancement strategy for extremely small targets is used, such as noise simulation, illumination transformation, contrast perturbation, etc., and the enhanced samples are input into the initial detection model. A refined sub-model with similar structure but higher resolution branches Secondary training is conducted in

[0128] Refined sub-model In the initial testing model Based on this, it is obtained by the following steps:

[0129] Copy the backbone and detection head to perform preliminary detection on the model The convolutional backbone (ResNet / CSP, etc.) and detection head (classification branch + regression branch) are used as templates to copy a new network structure.

[0130] Add a high-resolution feature pyramid and insert an extra ultra-high-resolution feature pyramid (such as or Downsampling) feature branch;

[0131] Fusion and output adjustment: the feature branch is input into the detection head together with the original multi-scale features, and the response to ultra-small targets is enhanced through additional channel fusion (or dedicated small target detection head).

[0132] Weight initialization, for the initial detection model For shared layers (backbone and regular modules), their trained weights are directly loaded; for newly introduced high-resolution branches and fusion layers, random initialization is performed using methods such as Xavier or He.

[0133] Fine-tuning is to use the hard-case image pool The hard examples (after enhancement) in the dataset are used as training data, the shared part is fine-tuned at a lower learning rate, and the high-resolution branches and fusion layers are optimized with a normal learning rate. At the same time, a strategy of freezing some deep parameters or unfreezing them in stages can be adopted to ensure that the refined branches converge quickly on the "hard-to-detect" samples without destroying the learned general features.

[0134] By the above method, the sub-model is refined The structure retains the detection capabilities that are highly consistent with the initial model, and the new high-resolution branch specifically enhances the perception of extremely small defects, so that the initial detection model can be more effectively strengthened after secondary training. Missed detection vulnerabilities in complex contexts;

[0135] During secondary training, reinforcement learning factors are introduced To highlight the contribution of difficult examples to model parameter updates:

[0136]

[0137] in: The objective function for the detection or classification layer, for example, an IoU-related loss or a sparsity-based segmentation penalty can be used; and They are actual labels and model predictions;

[0138] Based on the confidence of the candidate defect in the preliminary detection model Dynamically adjust weighting;

[0139] After training is completed, refine the sub-model It will be able to more accurately identify complex backgrounds and extremely small targets, compared with the preliminary detection model Form a complementary relationship.

[0140] Refine the sub-model Store as and preliminary detection model The parallel sub-models together form a system that can be integrated for reasoning in the next step (i.e., the fourth step), retaining the pool of difficult examples. , mark files and training logs for further incremental learning during subsequent online tuning or when adding new defect types;

[0141] During use, enhanced training is performed for difficult defects: a single complete training often ignores the particularity of low-confidence areas. By re-scanning and high-precision labeling these difficult examples, the model's adaptability to extreme scenarios is significantly improved.

[0142] The use of multi-scale resampling and reinforcement learning factors enables the network to have more accurate learning capabilities when facing defects with significant size changes or strong noise interference, greatly reducing the phenomenon of missing extremely small targets and refining the sub-model. Does not completely replace the initial detection model Instead, it takes its shortcomings and makes up for them to form a dual-model parallel structure, laying the foundation for subsequent multi-model fusion and online tuning.

[0143] Step 301 and step 302, through the closed-loop collaboration of hard case discovery, resampling, and enhanced training, significantly enhance the network's ability to distinguish small targets and complex backgrounds. If we only rely on the detection results of the preliminary detection model, it is easy to retain some extremely difficult to distinguish tiny defects without paying attention to them; if we only do hard case sampling without targeted secondary enhancement, it is not enough to significantly improve the detection level. By combining the two to produce a refined sub-model Focus on improving the initial detection model The shortcomings of the system can produce unexpected synergistic gain effects at the overall system level, forming a multi-round screening and optimization mechanism for extremely small target defects, which can greatly reduce the missed detection and false detection rates.

[0144] Step 4: When the production process requires taking the preliminary test model into account in actual reasoning Speed ​​and refinement of sub-models When the accuracy is high, the fusion strategy is adopted Combine IoU voting, comprehensive confidence evaluation, and background interference checks, and perform online dynamic tuning and secondary scanning on low-confidence areas, providing real-time feedback on new difficult examples and forming a continuous self-learning closed loop.

[0145] The step 4 includes the following contents:

[0146] Step 401: Multi-model fusion discrimination:

[0147] The two core detection models obtained from the third step: preliminary detection model and refined sub-models , and simultaneously predict defects on the same input image. Each of them outputs a series of bounding boxes, defect categories, and confidence scores, which are recorded as and ; This parallel reasoning can be achieved through multi-threading or GPU concurrency.

[0148] In order to combine the prediction results of the two models into a final output track, a fusion function is designed Merge and filter the detection results. In this step, factors such as IoU (Intersection over Union), confidence, and background interference are taken into consideration during fusion, so that the prediction results of different models for small defects complement each other. The fusion function The expression is as follows:

[0149]

[0150] in: It is a discrete set of all bounding boxes and their mutual IoU overlapping areas in the fusion process; It is a trainable or designable function that quantifies the overlap of bounding boxes detected by two models in the same area;

[0151] To comprehensively consider the confidence of the two models, if both are high, they will be further strengthened in the fusion result; if there are obvious differences, voting or other optimization methods can be adopted according to the usage scenario; Represents the defect confidence distribution output after the fusion of all detection results;

[0152] Through this fusion strategy, the preliminary detection model can be Provides stable detection of most common defects while utilizing refined sub-models Improved performance in accurate recognition of challenging areas;

[0153] Finally, in the fusion function Under the action of , to filter out frames with too low confidence or high overlap, and use the results for real-time difficult identification and secondary scanning.

[0154] When using, make full use of the respective advantages of the two-stage model: preliminary detection model Fast speed, good generalization, and refined sub-models It excels at targeting extremely difficult or tiny targets; the fused result is better than using either model alone in terms of overall accuracy and recall rate for difficult examples. If the confidence of the fusion result itself has been improved, the resource consumption in the downstream real-time secondary scanning and self-learning links can be saved, forming a synergistic effect of both efficiency and performance.

[0155] Step 402: Real-time identification of difficult samples and secondary scanning:

[0156] In the global detection results Based on the confidence level, for the confidence level below the confidence threshold Outputs from the , or defect patterns that are significantly abnormal compared to previous historical statistics, are all considered difficult samples and are added to the real-time scan list. Confidence threshold It can be updated online to balance detection speed and reliability. For example, if a large number of missed detections / false detections are detected on a batch of wafers, the specific threshold can be dynamically lowered. To capture more suspected defect areas.

[0157] Real-time scan list Perform secondary acquisition or high-magnification imaging on the defective area to obtain a clearer or more delicate local image; the obtained secondary scan image is input into the fusion detection process again, and the preliminary detection model can be skipped. , directly call the refined sub-model , to quickly locate the target area and overwrite the original suspected results with the updated detection results to improve accuracy.

[0158] If there are still defective areas that the system cannot determine after the second scan (such as extreme conflicts between two models), they will be recorded in the online difficult example database. , and temporarily adopt conservative strategies, such as manual review or higher-power microscopic review;

[0159] Online difficult case database The samples and scanning information in step 403 will be subsequently called by the dynamic tuning module or the deep training module again to achieve self-learning update.

[0160] When in use, rare or extreme situations can be detected again: Even if the fusion results show low confidence or high disagreement, the second scan ensures that these difficult defects are not directly ignored, thereby reducing missed detections and processing them immediately in the production process, improving the efficiency of capturing tiny defects and complex backgrounds, and further ensuring the yield rate.

[0161] Step 403: Online dynamic closed-loop tuning:

[0162] The online difficult example database collected in step 402 Included in the training pipeline along with manual or high-magnification microscopy final confirmation labels, and incrementally updated refined sub-models Or partial network weights of the two models; define the weight function of the online incremental learning strategy here , dynamically amplify or smooth the contribution of new samples, for example:

[0163]

[0164] in: represents the model parameters; A new loss or deviation measurement function may be developed that integrates multi-dimensional information, such as detection confidence, positioning error, defect category identification, etc.

[0165]

[0166] in: and are the model prediction box and the true box respectively; It is the generalized IoU, which measures the overlap and enclosing relationship between two boxes; For the true category The predicted probability of For the model The overall confidence of the candidate box;

[0167] is the weight coefficient of the three losses; , can be set as After cross-validation fine-tuning, is a nonlinear coupling index that amplifies the positioning, classification and confidence deviations respectively, Generally taken in The interval is used to differentiate the penalties for large errors and small errors.

[0168] The intensity of attention paid to difficult examples can be adjusted according to the final detection confidence; the fusion strategy can be automatically or semi-automatically updated according to the model performance after online incremental learning and confidence threshold ;

[0169] If the error rate decreases significantly over a period of time, the threshold for determining difficult samples can be appropriately increased. To reduce unnecessary secondary scanning; if new defect types emerge in large numbers, it is necessary to lower the judgment threshold of difficult samples And call more refined sub-models To ensure the recall rate.

[0170] This incremental learning and threshold adaptation process is continuously iterated during production. As new online difficult examples enter the system, the model continues to evolve, making it more adaptable to updated process lines, wafer manufacturing processes, and defect morphologies. By recording the incremental updates and indicator changes at each iteration, the long-term self-learning effectiveness of the entire system can be evaluated, allowing for manual intervention or rollback to a specific version of the model weights when necessary.

[0171] When used, compared with fixed model deployment, online dynamic tuning allows the detection system to continuously learn new defect patterns, helping to reduce the risk of increased false detection and missed detection rates due to process changes, raw material differences, etc.

[0172] Multi-model fusion provides a higher-confidence preliminary screening for detection outputs, real-time secondary scanning ensures that difficult defects are not missed, and online dynamic tuning allows the system to improve overall performance through continuous feedback and incremental learning. The self-iterating detection system has achieved improvements in accuracy and versatility.

[0173] Step 5: After obtaining multimodal information or time series images to track defect evolution, the aforementioned fusion features can be used to With the refined sub-model Adding cross-channel attention to the structure Aggregation functions with time series features , integrating multi-source factors to capture deep defect characteristics and conduct early warning of crack trends, expanding the boundaries of online tuning;

[0174] The step five includes the following:

[0175] Step 501: Multimodal data acquisition and preprocessing:

[0176] On the basis of single optical imaging, image data collected by sensors such as infrared or X-rays, as well as possible 3D shape measurement results, are added. These multimodal data are aligned with the existing optical images through spatial calibration and time stamps, and stored in the expanded data container. ; Each modality must retain a unique identifier (such as , , ) and optical images or Establish a traceable mapping to ensure that pixel coordinates or point cloud coordinates are correctly aligned during subsequent fusion;

[0177] In order to eliminate the differences in resolution, illumination or imaging principles between different modalities, a multimodal mapping function can be used according to the specific situation. Align the coordinates of infrared or X-ray images with optical images and unify them in grayscale or pseudo-color:

[0178]

[0179] in: Indicates the The original input of the modality; It is a new image after alignment that can be registered with the optical image; are the corresponding image coordinates; this function corrects the spatial deviation between different modalities through parameters that can be learned or manually calibrated;

[0180] In addition, for 3D topography data, the height or reflectivity information can be projected into a pseudo image of the same dimension and recorded in the data container. In the multimodal channel, under the premise of maintaining global coordinate consistency, the preliminary alignment and unified management of multimodal data are completed, laying the data foundation for subsequent multimodal feature fusion.

[0181] Step 502: Multimodal feature fusion and cross-channel attention:

[0182] Based on the network framework defined in the second step, an additional feature extraction branch is constructed for each new modality (the convolutional backbone network that is the same or similar to the original optical branch can be used) to form a multi-channel input; the multi-channel output feature set is recorded as , these multimodal features are mapped through cross-modal fusion Processing:

[0183]

[0184] Channel-level splicing + convolutional mapping, or a more advanced Cross-Attention structure, can be used to make different modalities complement each other in the channel dimension, achieving simultaneous capture of deep defects (such as internal cracks) and surface details (such as micropores);

[0185] To further emphasize the local areas most relevant to defects in multimodal data, a cross-channel attention module is introduced. , weight distribution of different modal information:

[0186]

[0187] in: Indicates feature concatenation or broadcast summation; It is a learnable attention generation function that weights the complementary or redundant components between modalities. Specifically, it usually includes: channel compression and expansion layers, such as a Convolution or fully connected layer, the concatenated multimodal features Mapping to a lower dimension; nonlinear activation (such as ReLU, Swish or GELU); channel remapping layer, and then another The convolution or linear layer restores the original number of channels; oftmax or Sigmoid normalization makes the output attention vector ( is the number of channels) can be directly used as the channel-by-channel weight;

[0188] Finally, these cross-modal attention factors are fused with the optical features to obtain multimodal cross-channel attention enhancement features. , output to the same inspection head or online tuning module as in steps 3 and 4, enabling multimodal integration within the established defect recognition process. Cross-channel attention methods fully exploit defect features that are hidden in infrared, X-ray, or 3D topography images but difficult to detect in optical images, thus addressing the limitations of the original approach, which primarily relied on optical textures.

[0189] Step 503: Timing information modeling and defect evolution tracking:

[0190] If multiple frames of time-series images can be obtained in a continuous shooting or real-time monitoring scenario, they will be stored in the data container in the order of timestamps. and mapped to a time series sample set with the original single frame annotation system , for the time series sample set Each time slice or batch frame in the process is marked with small defects and associated management, keeping the coordinates and defect ID mapping does not conflict;

[0191] In this time series environment, a time series convolution module (TConv) or sequence network (such as LSTM, time domain transformers) can be added after the multimodal feature output to integrate and predict the defect features of each frame in the time domain.

[0192] In order to capture the evolution trend of defects at different times, a time series feature aggregation function can be defined :

[0193]

[0194] in: Indicates at time Corresponding multimodal cross-channel attention enhancement features;

[0195] Mechanisms such as LSTM, GRU, or Transformers can be used to mine the dynamic changes of defect morphology (such as crack expansion and hole enlargement) from the time dimension.

[0196] Leveraging these time series features, predictive alerts can be implemented during the inspection or online tuning phase (step 4). Once the model detects a minor defect showing a significant deterioration trend over a short period of time, a higher-level review and maintenance process is automatically triggered to prevent the defect from causing serious failures in subsequent manufacturing processes or during final use. Furthermore, time series data can be recorded in a database of difficult examples for further focused learning using a similar reinforcement strategy as in step 3.

[0197] Unlike the previous single-frame static detection, this step can detect early cracks or potential hazards in the time series, improving the overall foresight of inspection and maintenance. Real-time observation of defect change rates or trends provides a strong basis for dynamic threshold adjustment and model update decisions, enabling adaptive parameter and model migration in new process environments.

[0198] This solution elevates the perception and tracking of extremely small defects to a multi-dimensional and dynamic level, enabling more proactive monitoring of hidden internal defects and early evolution trends, and filling the blind spots of single-view optical images. Simply adding modules to the optical image detection system would make it difficult to effectively capture the characteristics of hidden or dynamic defects, further strengthening the integrity and reliability of the small defect detection process.

[0199] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only for some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0202] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0203] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A deep learning-based small target defect recognition model training method, characterized by: include, High-resolution sampling and pixel-level annotation are applied to real and synthetic image sets. Multi-channel label mapping is generated through unified ID management and high-resolution preprocessing to build initial data that encompasses the diversity of rare and small defects. Multi-scale feature fusion and spatial attention mechanisms are introduced to the image features in the storage container to significantly amplify fine-grained targets at the fused feature level and ensure that multi-layer semantics and edge details form a learnable context; Based on the preliminary detection model, the image is inferred. If a candidate defect area with low confidence or high background interference appears, the image corresponding to the candidate list is resampled and stored in the difficult example set. The refined sub-model is generated by combining reinforcement learning factor training; A fusion strategy is used to cross-check IoU voting, comprehensive confidence evaluation, and background interference. Online dynamic tuning and secondary scanning are performed on low-confidence areas, providing real-time feedback on new difficult examples and enabling continuous self-learning. After acquiring multimodal information or time-series images to track defect evolution, cross-channel attention and time-series features are added to the fusion feature and refinement sub-model to fuse multi-source elements to capture deep defect characteristics and provide early warning of crack trends.

2. The method for training a small target defect recognition model according to claim 1, characterized in that: Obtain high-resolution images from wafers and assign labels to them. Generative adversarial networks are then used to embed defect templates into actual wafer backgrounds or random textures to generate synthetic data. All real and synthetic data are saved in a data container, and the defect coverage calculated in real time is written into the metadata file together with the original resolution information of each image.

3. The method for training a small target defect recognition model according to claim 2, wherein: After two rounds of independent annotation for the same defect, the annotation results are checked for consistency, and the consistency score is used to measure the overlap between different annotation rounds; If the consistency score is greater than the preset consistency threshold, the annotation is accepted; otherwise, the corresponding defect is included in the abnormal annotation list, triggering the re-annotation or expert review process, obtaining the defect information and updating the storage container.

4. The method for training a small target defect recognition model according to claim 3, wherein: The labeled image and defect information are input into the basic backbone of the convolutional neural network to obtain shallow and deep features. A layer fusion module is introduced to upsample or downsample the shallow and deep features and align them. The fused feature map is obtained after splicing or weighted summation. During the fusion, a multi-scale weighting function is defined to perform nonlinear adjustment on the features of each scale.

5. The method for training a small target defect recognition model according to claim 4, wherein: The fused feature map is input into the spatial attention module to learn trainable attention maps for different regions in the image, and is multiplied pixel by pixel with the fused feature map to generate spatial attention enhanced features; During the training phase, the internal parameters of the global memory module and activation function are continuously corrected. When the attention module, detection head and backbone network are trained together, the preliminary detection model is obtained.

6. The method for training a small target defect recognition model according to claim 5, wherein: Large-scale wafer surface images are input for inference, and the preliminary detection model outputs a confidence value for each candidate defect and filters out low-confidence defects using a global judgment threshold. The background confusion degree is used to measure the interference strength between the fused features of the candidate defect area. When the background confusion degree exceeds the preset confusion threshold, the corresponding candidate defect area is marked as a background easily confused area, and a candidate list consisting of overlapping low-confidence defects and background easily confused areas is output.

7. The method for training a small target defect recognition model according to claim 6, wherein: Perform high-resolution resampling or local high-magnification scanning on the image areas marked as easily confused by the background or low-confidence defects in the candidate list to obtain clearer sub-images and store them in the hard-to-remember image pool; The difficult example images are annotated again at the pixel level and an annotation file is generated. A special data enhancement strategy for extremely small objects is adopted, and a reinforcement learning factor is introduced to perform secondary training on the refined sub-model.

8. The method for training a small target defect recognition model according to claim 7, wherein: The detection frames of the preliminary detection model and the refined sub-model are subjected to IoU voting and confidence synthesis through a fusion function to obtain a global detection result, thereby filtering out frames with too low confidence or high overlap, and using the result for real-time difficult identification and secondary scanning.

9. The method for training a small target defect recognition model according to claim 8, wherein: Outputs with confidence levels lower than the confidence threshold are identified as problematic samples and added to the real-time to-be-scanned list; Perform secondary acquisition or high-magnification imaging on the defect areas in the real-time waiting-to-scan list. The obtained secondary scan images are input into the fusion detection process again, and the updated detection results overwrite the original suspected results. If there are still defective areas that the system cannot determine after the second scan, they will be recorded in the online difficult example database and a conservative strategy will be adopted temporarily.

10. The method for training a small target defect recognition model according to claim 9, characterized in that: Incorporate an online database of hard examples and final labels confirmed manually or by high-magnification microscopy into the training pipeline, and use a weight function to incrementally update the weights of the refined sub-model or part of the network of the two models; Based on the model performance after online incremental learning, the fusion strategy and confidence threshold are automatically or semi-automatically updated.

11. The method for training a small target defect recognition model according to claim 10, wherein: The newly added multimodal image data and 3D shape measurement results are stored in the expanded data container through spatial calibration and timestamp alignment; A multimodal mapping function is used to align the coordinates of the infrared or X-ray image with the optical image, and to unify the grayscale or pseudo-color modes.

12. The method for training a small target defect recognition model according to claim 11, characterized in that: An additional feature extraction branch is constructed for each new modality to form a multi-channel input, which is processed through cross-modal fusion mapping. After weight distribution of different modal information of the cross-channel attention module, the multimodal cross-channel attention enhanced feature is obtained by fusion.

13. The method for training a small target defect recognition model according to claim 12, wherein: After the multimodal feature output, a temporal convolution module is added to integrate and predict the defect features of each frame in the time domain, and the defect evolution trend score is calculated using the temporal feature aggregation function. If the obtained trend score exceeds the corresponding alarm threshold, a higher level review and maintenance process will be automatically triggered.

Images