Intelligent detection method and system for surface defects of micro-nano structure

Abstract

The invention relates to the technical field of defect detection methods, in particular to a micro-nano structure surface defect intelligent detection method and system, and the method comprises the steps: controlling an automatic sample table to enable a carried sample to move in a three-dimensional direction, and obtaining a plurality of small-range microspur images; splicing the plurality of small-range micro-distance images to form a high-resolution image; based on the high-resolution image, performing analysis through a deep learning convolutional neural network model, and identifying defects on the surface of the micro-nano structure; according to an identification result, generating a test case set, and simulating images acquired on the surface of the micro-nano structure by different sampling parameters; calculating performance parameter indexes of the system, and evaluating feasibility of the system; the high-resolution image is subjected to digital processing, defect detection on the surface of the micro-nano structure is achieved, the automatic high-resolution image collecting and splicing technology is provided, manual intervention is greatly reduced, and the data obtaining efficiency and quality are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection methods, and more specifically, to an intelligent detection method for surface defects of micro-nano structures and a system thereof. Background Art

[0002] With the rapid development of micro-nano manufacturing technology, the quality control of micro-nano structure surface is becoming more and more important. Micro-nano structure surface defects not only affect the performance and reliability of the product, but may also lead to a decrease in the efficiency of the entire production line. Therefore, efficient and accurate micro-nano structure surface defect detection technology has become a key demand in the manufacturing industry.

[0003] Traditional methods for detecting defects on micro-nanostructure surfaces mainly rely on manual visual inspection or simple machine vision technology. Although manual visual inspection can identify complex defect patterns, it has problems such as low efficiency, strong subjectivity, and easy fatigue. Although simple machine vision technology improves detection efficiency, its defect recognition ability in complex backgrounds is limited and it is difficult to cope with diverse micro-nanostructure surfaces.

[0004] In recent years, with the rise of deep learning technology, some researchers have tried to apply convolutional neural networks to surface defect detection of micro-nano structures. These methods have achieved certain results in specific scenarios, but there are still some significant limitations. First, existing deep learning methods often require a large amount of labeled data, and in the field of micro-nano manufacturing, obtaining large-scale high-quality defect samples is a huge challenge. Secondly, these methods are usually optimized for specific types of micro-nano structures or defects, lacking versatility and scalability. Furthermore, existing methods have high computational complexity when processing high-resolution images, making it difficult to meet the needs of real-time detection.

[0005] In addition, existing micro-nanostructure surface defect detection systems often operate independently and lack effective model updating and knowledge sharing mechanisms. This makes it difficult for the system to adapt to new defect types or different production conditions, and its performance may gradually deteriorate after long-term use.

[0006] Finally, existing technologies perform poorly when dealing with extreme surface conditions, such as highly nonlinear or irregular surfaces, which limits their application in advanced manufacturing.

[0007] In view of the above problems, there is an urgent need for new intelligent detection methods and systems for micro-nanostructure surface defects that can overcome the limitations of existing technologies and achieve efficient, accurate and universal defect detection. Summary of the invention

[0008] In view of these problems, the present invention proposes a micro-nano structure surface defect intelligent detection method and system thereof. The present invention aims to solve the problems existing in the existing micro-nano structure surface defect detection technology, such as difficulty in data acquisition, weak model generalization ability, insufficient real-time performance, poor system scalability, and insufficient adaptability to extreme conditions.

[0009] The present invention provides an intelligent detection method for micro-nano structure surface defects, comprising:

[0010] The acquisition steps include:

[0011] Control the automated sample stage to move the sample in three dimensions and obtain multiple small-range macro images;

[0012] splicing the multiple small-range macro images to form a high-resolution image;

[0013] Processing steps include:

[0014] Based on the high-resolution image, the defects on the surface of the micro-nano structure are identified by analyzing the image through a deep learning convolutional neural network model;

[0015] Based on the recognition results, a test case set is generated to simulate the images obtained on the surface of micro-nano structures with different sampling parameters;

[0016] Output steps include:

[0017] Calculate the performance parameters of the system and evaluate the feasibility of the system;

[0018] The high-resolution image is digitally processed to achieve defect detection on the surface of the micro-nano structure.

[0019] Preferably, the obtaining step specifically includes:

[0020] Control the three-dimensional motion platform to lift and lower vertically, and perform macro alignment between the microscope and the sample;

[0021] Scan the sample through the optical system module to obtain multiple small-range images;

[0022] The multiple small-range images are registered and a homography matrix is ​​calculated, a perspective transformation is performed on the homography matrix, and the multiple small-range images after the perspective transformation are spliced.

[0023] Preferably, the convolutional neural network model in the processing step is an Inception-v5 structure improved based on the Inception structure, including:

[0024] The first convolution module includes a 3×3 convolution layer, a 3×3 convolution layer, and a 3×3 convolution layer;

[0025] Sampling module, including global pooling layer and fully connected layer;

[0026] The second convolution module includes a 5×5 convolution layer, a 3×3 convolution layer, a 3×3 convolution layer, and a 3×3 convolution layer;

[0027] The Inception-v5 structure module includes 1×1 convolution layer, 1×3 convolution layer, 3×1 convolution layer, 3×3 convolution layer, average pooling layer, 1×1 convolution layer, 1×3 convolution layer, 3×1 convolution layer and 3×3 convolution layer.

[0028] Preferably, the processing step further comprises:

[0029] Setting hyperparameters for the convolutional neural network model, including learning rate and learning rate decay;

[0030] Execute the training and parameter adjustment of the convolutional neural network model, stop the training when the loss function drops to close to 0, and obtain a preliminarily trained network model;

[0031] Wherein, the loss function is:

[0032]

[0033] Among them, L i j represents the loss function, y i represents the probability that the i-th pixel is the defect class, 1 represents the probability that the pixel is the background class, N represents the number of pixels, and i and j represent the width and length of the image respectively.

[0034] Preferably, the processing step further comprises:

[0035] Using a grid search method, the learning rate and the learning rate decay are adjusted to obtain multiple groups of performance parameters;

[0036] Using an algorithm based on linear regression analysis, fitting an optimal learning rate and a learning rate decay according to the multiple sets of performance parameters;

[0037] The optimal learning rate and the learning rate decay value are input into the initially trained network model to obtain a trained network model.

[0038] Preferably, the output step further comprises:

[0039] Replace y and y′ in the loss function with y>a·y′ and input into the trained network model, where y represents the true label, i.e., whether the image pixel is a defect; y′ represents whether the system output pixel is a defect; a is a preset trainable parameter;

[0040] Calculate L(y, y′) = |ya·y′| to obtain the optimized network model;

[0041] The parameters of the optimized network model are set to include: a first training data set, a second training data set and a third training data set, and a training program is executed to obtain a first training model, a second training model and a third training model respectively.

[0042] Preferably, the output step further comprises:

[0043] Input the test data set into the first training model, the second training model and the third training model to verify the detection accuracy results of the three, and save the first training model, the second training model and the third training model with high accuracy;

[0044] The test data set includes: a non-intersecting data portion of the first training data set, the second training data set, and the third training data set.

[0045] Preferably, the processing step further comprises the following image preprocessing step:

[0046] Using a filter to remove noise, the filter is at least one of a Gaussian filter, a median filter, a mean filter or a Kalman filter;

[0047] Use edge detection algorithms to extract edge information in images to enhance the contrast of defective areas;

[0048] Improve the contrast of defective areas through contrast enhancement technology, using histogram equalization or local contrast enhancement methods;

[0049] Use sharpening and deblurring algorithms to improve image sharpness for better identification of tiny defects;

[0050] Correct the image distortion.

[0051] As an advantage, the method further includes a boundary adaptability test step:

[0052] Testing of extreme surface conditions, including extreme surface textures, nonlinearities, and irregularities;

[0053] Verify the system's defect detection performance and evaluate the system's robustness and adaptability.

[0054] Preferably, the image acquisition module is used to control the automated sample stage to move the sample carried by the stage in three dimensions, obtain multiple small-range macro images, and stitch the macro images to form a high-resolution image;

[0055] An image analysis module, used to build a convolutional neural network model based on deep learning, analyze the high-resolution image, and identify defects on the surface of the micro-nano structure;

[0056] A test module, used to generate a test case set, simulate images obtained on the surface of the micro-nano structure with different sampling parameters, calculate the performance parameter index of the system, and evaluate the feasibility of the system;

[0057] An image processing module, used for digitally processing the high-resolution image to realize defect detection on the surface of the micro-nano structure;

[0058] The convolutional neural network model in the image analysis module adopts the Inception-v5 structure improved based on the Inception structure, including a first convolution module, a sampling module, a second convolution module and an Inception-v5 structure module;

[0059] The system also includes a cloud deployment module for packaging the trained model into a file and uploading it to the cloud to achieve centralized management and flexible deployment of the model;

[0060] The system also includes a real-time online learning module for receiving new image data for real-time testing, and automatically updating model parameters according to the test results to achieve continuous optimization and improved adaptability of the model.

[0061] The present invention innovatively combines high-precision image acquisition, multi-scale deep learning models, adaptive optimization strategies, and cloud deployment technologies to achieve a comprehensive and efficient intelligent detection method and system for micro-nanostructure surface defects. This method not only significantly improves the accuracy and efficiency of defect detection, but also has strong versatility and scalability.

[0062] Specifically, the beneficial effects of the present invention are mainly reflected in the following aspects:

[0063] First, the automated high-resolution image acquisition and stitching technology proposed in this paper greatly reduces manual intervention and improves the efficiency and quality of data acquisition. This not only reduces the reliance on large-scale annotated data, but also provides high-quality input for subsequent deep learning analysis.

[0064] Secondly, the convolutional neural network model based on the improved Inception-v5 structure has a powerful multi-scale feature extraction capability and can effectively capture defect features of different types and sizes. This design greatly improves the versatility of the model, enabling it to adapt to various micro-nanostructure surfaces and defect types.

[0065] Furthermore, the adaptive learning rate adjustment strategy and innovative loss function design proposed in the present invention significantly improve the training efficiency and performance of the model, which not only accelerates the convergence process of the model, but also enhances the model's ability to identify different types of defects.

[0066] In addition, the multi-model training and integration strategy of the present invention further improves the robustness and generalization ability of the system. By training and integrating multiple models, the system can better handle complex and diverse defect situations and reduce the deviation that may be caused by a single model.

[0067] The invention also introduces a boundary adaptability test, which ensures the reliability and adaptability of the system in practical applications by verifying the system performance under extreme surface conditions. This greatly expands the application scope of the system and enables it to cope with more complex and changeable manufacturing environments.

[0068] Finally, the cloud deployment and real-time online learning mechanism of the present invention solves the problems of poor scalability and difficulty in continuous optimization of traditional systems. This design enables the system to continuously learn and adapt to new defect types and surface features, maintain long-term high performance, and also facilitates model sharing and updating.

[0069] In summary, the micro-nano structure surface defect intelligent detection method and system provided by the present invention not only overcomes the limitations of the prior art, but also achieves significant improvements in detection accuracy, efficiency, versatility, scalability and adaptability. This provides a powerful and flexible solution for quality control in the field of micro-nano manufacturing, and is expected to promote technological progress and efficiency improvement in related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Flow chart of the method of the present invention.

[0071] Figure 2 It is a structural diagram of the convolutional neural network model of the present invention.

[0072] Figure 3 This is a flowchart of image preprocessing of the present invention. DETAILED DESCRIPTION

[0073] Please refer to Figure 1-3 The present invention provides a method and system for intelligent detection of surface defects of micro-nano structures. The method uses advanced image processing and deep learning technology to achieve high-precision automatic detection of surface defects of micro-nano structures.

[0074] First, the method includes an acquisition step. In this step, the sample is moved in three dimensions by controlling the automated sample stage to acquire multiple small-range macro images. Preferably, the automated sample stage can be precisely controlled in the X, Y, and Z directions, with a movement accuracy of up to 0.1 μm, to ensure the acquisition of high-quality macro images. For example, for a 1 cm × 1 cm micro-nano structure sample, it can be divided into 100 × 100 small areas, and a high-resolution macro image is acquired for each area.

[0075] Subsequently, the method stitches the acquired multiple small-range macro images to form a high-resolution image. During the stitching process, an image stitching algorithm based on feature point matching, such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) algorithm, is preferably used to ensure the accuracy of the stitching. The resolution of the stitched high-resolution image can reach 10nm / pixel, which can clearly present the subtle features of the surface of the micro-nano structure.

[0076] In the processing step, the method of the present invention analyzes the high-resolution image through a deep learning convolutional neural network model to identify defects on the surface of the micro-nano structure. In one embodiment of the present invention, a convolutional neural network model with an improved Inception-v5 structure is used. The model has multi-scale feature extraction capabilities and can effectively capture defect features of different scales.

[0077] Specifically, the convolutional neural network model of the present invention includes the following modules:

[0078] 1. The first convolution module: contains three 3×3 convolutional layers. These convolutional layers are used to extract low-level features of the image, such as edges, textures, etc. Each convolutional layer is followed by a Batch Normalization layer and a ReLU activation function to accelerate network convergence and enhance nonlinear expression capabilities.

[0079] 2. Sampling module: includes global pooling layer and fully connected layer. The global pooling layer performs global average pooling on the feature map to reduce the number of parameters and retain global information. The fully connected layer is used for feature fusion and dimensionality transformation.

[0080] 3. The second convolution module: contains one 5×5 convolution layer and three 3×3 convolution layers. These convolution kernels of different sizes can capture features of different scales and enhance the multi-scale expression ability of the model.

[0081] 4. Inception-v5 structural module: It contains multiple parallel convolution branches, such as 1×1 convolution, 1×3 convolution, 3×1 convolution, 3×3 convolution, etc., as well as average pooling layer. This structural design can extract multi-scale features at the same time and improve the model's ability to recognize defects of different sizes.

[0082] In a preferred embodiment of the present invention, the specific parameters of the convolutional neural network model are set as follows:

[0083] The number of convolution kernels in the 3×3 convolution layer in the first convolution module is 64, 128, and 256 respectively;

[0084] The number of neurons in the fully connected layer in the sampling module is 1024;

[0085] The number of convolution kernels in the 5×5 convolution layer in the second convolution module is 384, and the number of convolution kernels in the three 3×3 convolution layers are 192, 224, and 256 respectively;

[0086] The number of convolution kernels in each convolutional layer in the Inception-v5 structure module is dynamically adjusted according to the size of the input feature map, generally between 64 and 384;

[0087] Based on the recognition results, the method generates a set of test cases to simulate images obtained on the surface of micro-nano structures with different sampling parameters. The purpose of this step is to evaluate the performance of the system under different sampling conditions. For example, different scanning speeds (such as 0.1μm / s to 10μm / s), different lighting conditions (such as bright field, dark field, differential interference contrast, etc.), different focal plane positions (such as ±0.5μm to ±5μm) and other parameter combinations can be simulated to generate a series of test images.

[0088] In the output step, the method calculates the performance parameter indicators of the system and evaluates the feasibility of the system. Commonly used performance indicators include accuracy, precision, recall and F1 score. In one embodiment of the present invention, the evaluation criteria of system performance are as follows:

[0089] Accuracy > 95%;

[0090] Accuracy > 90%;

[0091] Recall > 90%

[0092] F1 score > 0.90;

[0093] The calculation formulas for these indicators are as follows:

[0094]

[0095] Among them, TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

[0096] Finally, the method performs digital processing on the high-resolution image to realize defect detection on the surface of the micro-nano structure. The digital processing includes operations such as image enhancement, noise removal, and edge detection to improve the accuracy and reliability of defect detection.

[0097] In another embodiment of the present invention, the acquisition step specifically includes the following sub-steps: First, the three-dimensional motion platform is controlled to perform vertical lifting and lowering to perform macro alignment on the microscope and the sample. This process usually uses an autofocus algorithm, such as a climbing algorithm based on image clarity or a phase-based focusing algorithm, to achieve precise focusing. The focusing accuracy can reach ±0.1μm, ensuring the acquisition of a clear macro image.

[0098] Then, the sample is scanned by the optical system module to obtain multiple small-range images. The optical system module can include a high-power objective lens (such as a 100x oil objective), an illumination system (such as LED illumination or laser illumination), and a high-resolution CCD or CMOS camera. During the scanning process, step scanning or continuous scanning mode can be used, and the scanning speed and step length can be adjusted according to the sample characteristics and requirements.

[0099] Next, the multiple small-scale images are registered and the homography matrix is ​​calculated, the homography matrix is ​​subjected to perspective transformation, and the multiple small-scale images after the perspective transformation are spliced. During the image registration process, a feature point matching algorithm such as SIFT or ORB (Oriented FAST and Rotated BRIEF) can be used to extract image feature points and match them. The calculation of the homography matrix can use the RANSAC (Random Sample Consensus) algorithm to improve robustness. Perspective transformation and image splicing can be implemented using open source libraries such as OpenCV.

[0100] In the method of the present invention, the convolutional neural network model used in the processing step is the Inception-v5 structure improved based on the Inception structure. The design concept of this structure is to improve the model's ability to recognize defects of different scales through multi-scale feature extraction and feature fusion. The Inception-v5 structure includes the following key modules:

[0101] 1. The first convolution module: contains three 3×3 convolutional layers. These convolutional layers are used to extract low-level features of the image, such as edges, textures, etc. The number of output channels of each convolutional layer can be increased layer by layer, such as 64, 128, 256, to gradually increase the complexity of the features.

[0102] 2. Sampling module: includes global pooling layer and fully connected layer. The global pooling layer performs global average pooling on the feature map, which can effectively reduce the number of parameters and retain global information. The number of neurons in the fully connected layer can be set to 1024 or 2048 for feature fusion and dimensionality transformation.

[0103] 3. The second convolution module: contains one 5×5 convolution layer and three 3×3 convolution layers. The 5×5 convolution layer can capture features with a larger receptive field, while the 3×3 convolution layer can extract finer local features. The number of output channels of these convolution layers can be set to 384, 192, 224, 256, etc.

[0104] 4. Inception-v5 structure module: This is the core module of the network, which contains multiple parallel convolution branches. Specifically including:

[0105] 1×1 convolutional layer: used for dimensionality reduction and cross-channel information integration;

[0106] 1×3 convolution layer and 3×1 convolution layer: used to capture features in different directions;

[0107] 3×3 convolutional layer: used to extract local features;

[0108] Average pooling layer: used to retain global information;

[0109] 1×1 convolutional layer: used for dimensionality reduction;

[0110] 1×3 convolution layer and 3×1 convolution layer: used to further extract features in different directions;

[0111] 3×3 convolutional layer: used to extract features with a larger receptive field;

[0112] This multi-branch parallel structural design allows the network to extract multi-scale features simultaneously, greatly improving the model's ability to identify defects of different sizes and shapes. In practical applications, the specific parameters of each module can be fine-tuned according to the specific surface characteristics of the micro-nano structure and the defect type to obtain the best detection effect.

[0113] Through the above detailed description, the intelligent detection method for micro-nanostructure surface defects of the present invention realizes high-precision and high-efficiency defect detection. The method combines advanced image processing technology and deep learning algorithm, can adapt to different types of micro-nanostructure surfaces, and has strong versatility and practical value. In a preferred embodiment of the present invention, the processing step also includes setting hyperparameters for the convolutional neural network model, including learning rate and learning rate decay. The selection of these hyperparameters has an important influence on the training effect of the model. Typically, the initial learning rate can be set between 0.001 and 0.1, and the learning rate decay can adopt an exponential decay or step decay strategy. For example, the learning rate can be reduced to 0.1 times the original every 10 epochs.

[0114] The method of the present invention performs training and parameter adjustment of a convolutional neural network model, and stops training when the loss function drops to near 0, thereby obtaining a preliminarily trained network model. In practical applications, a threshold value may be set, such as stopping training when the loss function value is less than 0.01 or when the change in the loss function value for five consecutive epochs is less than 0.001. This strategy can effectively prevent overfitting while ensuring that the model achieves better performance.

[0115] It is worth noting that the present invention adopts a special loss function, whose mathematical expression is as follows:

[0116]

[0117] Among them, L i j represents the loss function, y i represents the probability that the i-th pixel is the defect class, 1 represents the probability that the pixel is the background class, N represents the number of pixels, and i and j represent the width and length of the image respectively.

[0118] This loss function is actually a pixel-level cross entropy loss, which can effectively deal with the problem of category imbalance and is particularly suitable for scenarios such as micro-nanostructure surface defect detection where the ratio of positive and negative samples may be seriously unbalanced.

[0119] In another embodiment of the present invention, the processing step also includes an adaptive learning rate adjustment strategy. First, the learning rate and learning rate decay are adjusted using a grid search method to obtain multiple groups of performance parameters. The range of the grid search can be set to: the learning rate is from 0.0001 to 0.1, and the logarithmic scale is uniformly sampled with a base of 10; the learning rate decay factor is from 0.1 to 0.9, and the linear uniform sampling is performed. Each group of parameters is trained and verified, and the performance indicators of the model, such as accuracy, recall, etc., are recorded.

[0120] Subsequently, the method uses an algorithm based on linear regression analysis to fit the optimal learning rate and learning rate decay according to the multiple sets of performance parameters. Specifically, a multivariate linear regression model can be established by using performance indicators (such as F1 scores) as dependent variables and learning rates and learning rate decay as independent variables. By analyzing the regression coefficient and the coefficient of determination, the parameter combination that has the greatest impact on the model performance can be found.

[0121] Finally, the optimal learning rate and learning rate decay value are input into the initially trained network model to obtain a trained network model. This adaptive learning rate adjustment strategy can quickly find a near-optimal hyperparameter combination in a wide range, significantly improving the efficiency of model training and final performance.

[0122] In an innovative embodiment of the present invention, the output step also includes a loss function optimization strategy. The strategy replaces y and y′ in the loss function with y>a·y′ and inputs it into the trained network model. Among them, y represents the true label, that is, whether the image pixel is a defect; y′ represents whether the system output pixel is a defect; a is a preset trainable parameter. Preferably, the initial value of a can be set to 1 and optimized during the training process.

[0123] Then, this method calculates the optimized loss function:

[0124] L(yy′)=|ya·y′|,

[0125] The advantage of this loss function design is that by introducing a trainable parameter a, the model's adaptability to different types of defects is increased. Parameter a can be understood as a scaling factor that can adaptively adjust the model's sensitivity to defects and background. For example, when a>1, the model will be more inclined to classify pixels as defects, which is particularly useful when detecting tiny defects; when a<1, the model will judge defects more conservatively, which helps reduce false positives.

[0126] In addition, this optimization strategy allows the model to dynamically adjust parameter a during training to find the optimal solution, thereby better balancing the precision and recall of defect detection. This approach is of great significance for dealing with defects of different natures and improving the overall performance of the system. In practical applications, such a mechanism can help improve the accuracy of surface defect detection of micro-nano structures under complex backgrounds, while adjusting the detection threshold according to the requirements of different application scenarios.

[0127] After obtaining the optimized network model, the method of the present invention sets the parameters of the optimized network model to include: a first training data set, a second training data set and a third training data set, and executes a training program to obtain a first training model, a second training model and a third training model respectively. The purpose of this multi-model training strategy is to improve the robustness and generalization ability of the system.

[0128] Specifically, the total training data can be randomly divided into three data sets in a ratio of 6:2:2. The first training data set contains 60% of the data and is used to train the main model; the second and third training data sets each contain 20% of the data and are used to train the auxiliary model. These three models can use the same network structure, but through training with different data sets, they can capture different features and patterns in the data.

[0129] In a preferred embodiment of the present invention, the output step also includes model integration and performance verification. Specifically, the test data set is input into the first training model, the second training model and the third training model to verify the detection accuracy results of the three, and the first training model, the second training model and the third training model with high accuracy are saved.

[0130] The test data set here refers to an independent data set that is completely different from the training data set, which is used to fairly evaluate the generalization ability of the model. Preferably, the size of the test data set can be set to about 20% of the total data volume. It is worth noting that the test data set includes: the data part of the first training data set, the second training data set and the third training data set without intersection, which ensures the fairness and effectiveness of the test.

[0131] In practical applications, an integrated learning strategy can be used to comprehensively utilize the advantages of these three models. For example, the soft voting method can be used to weight the output probabilities of the three models to obtain the final prediction result. The weights can be dynamically adjusted based on the performance of each model on the validation set. This integrated strategy can effectively reduce the deviation of a single model and improve the overall prediction accuracy and stability.

[0132] Through the above detailed description, the intelligent detection method for surface defects of micro-nano structures of the present invention realizes the adaptive optimization and integrated learning of the model, greatly improving the accuracy and robustness of the detection. This method is not only suitable for conventional micro-nano structure surface defect detection, but also can well handle complex and diverse defect types, and has broad application prospects. In a preferred embodiment of the present invention, the processing step also includes a series of image preprocessing operations, aiming to improve the accuracy and reliability of subsequent defect detection. First, the method uses a filter to remove noise. Preferably, at least one of a Gaussian filter, a median filter, a mean filter or a Kalman filter can be selected. For example, for Gaussian noise, a Gaussian filter can be used, and its kernel size can be set to 3×3 or 5×5, and the standard deviation σ can be adjusted according to the degree of noise, usually between 0.5 and 2; for salt and pepper noise, the median filter works better, and a window size of 3×3 or 5×5 can be selected.

[0133] Next, the method of the present invention uses an edge detection algorithm to extract edge information in the image to enhance the contrast of the defect area. Commonly used edge detection algorithms include Sobel operator, Canny operator and Laplacian operator. In micro-nano structure surface defect detection, the Canny edge detection algorithm usually performs better because it can effectively suppress noise and detect real weak edges. When using the Canny operator, the low threshold can be set to 1 / 2 or 1 / 3 of the high threshold, for example, the low threshold is 50 and the high threshold is 150.

[0134] Subsequently, the method improves the contrast of the defective area by contrast enhancement technology. Preferably, histogram equalization or local contrast enhancement method can be used. Histogram equalization is suitable for images with low overall contrast, while for images with obvious local features such as micro-nano structure surfaces, local contrast enhancement method may be more effective. For example, the adaptive histogram equalization (CLAHE) algorithm can be used to divide the image into 8×8 or 16×16 small blocks, histogram equalization is performed on each small block, and then bilinear interpolation is used to merge the processing results.

[0135] The method of the present invention also uses a sharpening algorithm and a deblurring algorithm to improve the sharpness of the image so as to better identify tiny defects. Commonly used sharpening algorithms include Unsharp Mask and Laplacian sharpening. For example, when using UnsharpMask, the radius can be set to 1-2 pixels, the amount can be 100% to 200%, and the threshold can be set to 3-5 levels. The deblurring algorithm can select Wiener filtering or Lucy-Richardson algorithm, which can effectively reduce the effects of motion blur or defocus blur.

[0136] Finally, the method performs distortion correction on the image. This step is critical to ensure measurement accuracy, especially when using a high-magnification microscope. Distortion correction can be performed using a checkerboard calibration plate to calibrate the camera, obtain the camera's intrinsic parameters and distortion coefficients, and then use these parameters to correct the image.

[0137] In another innovative embodiment of the present invention, a boundary adaptability test step is introduced. The purpose of this step is to evaluate the performance of the system under extreme conditions and ensure the robustness of the system in practical applications. Specifically, the method tests extreme surface conditions, including extreme surface textures, nonlinearities, and irregularities.

[0138] For testing of extreme surface textures, a series of samples with different roughness can be prepared, ranging from nanometers to micrometers. For example, samples with Ra (arithmetic mean deviation) values ​​ranging from 1nm to 10μm can be used. Nonlinear testing can include surfaces with sudden slopes, such as parabolic or hyperbolic structures. Irregularity testing can use surfaces with randomly distributed bumps and depressions.

[0139] Under these extreme conditions, this method verifies the system's defect detection performance and evaluates the system's robustness and adaptability. Evaluation indicators can include detection accuracy, missed detection rate, and false detection rate. For example, it can be set that under extreme conditions, the detection accuracy is not less than 90%, and the missed detection rate and false detection rate are both no more than 5%. If the system can meet these indicators, it is considered to have good boundary adaptability.

[0140] Finally, the present invention also provides a micro-nanostructure surface defect intelligent detection system corresponding to the above method. The system includes multiple functional modules, each of which is carefully designed to achieve a specific function.

[0141] The image acquisition module 1 is used to control the automated sample stage to move the sample carried by it in three dimensions, obtain multiple small-range macro images, and stitch the macro images to form a high-resolution image. The module may include a high-precision three-axis mobile platform, a high-resolution camera, and dedicated image stitching software.

[0142] Image analysis module 2 is used to build a convolutional neural network model based on deep learning, analyze the high-resolution image, and identify defects on the surface of the micro-nano structure. The core of this module is the convolutional neural network of the Inception-v5 structure described above, which runs on a GPU to accelerate the calculation process.

[0143] The test module 3 is used to generate a test case set, simulate images obtained on the surface of the micro-nano structure with different sampling parameters, calculate the performance parameter index of the system, and evaluate the feasibility of the system. This module can include two submodules: parameter generator and performance evaluator.

[0144] The image processing module 4 is used to digitally process the high-resolution image to realize defect detection on the surface of the micro-nano structure. This module realizes the various image preprocessing algorithms described above.

[0145] It is worth noting that the system also includes a cloud deployment module 5, which is used to encapsulate the trained model into a file and upload it to the cloud to achieve centralized management and flexible deployment of the model. This cloud deployment strategy allows the system to easily share and update models between different detection sites, greatly improving the scalability and maintainability of the system.

[0146] In addition, the system innovatively introduces a real-time online learning module 6, which is used to receive new image data for real-time testing, automatically update model parameters according to the test results, and achieve continuous optimization and adaptability of the model. This module enables the system to continuously learn and adapt to new defect types and surface features, maintaining long-term high performance.

[0147] Through the above detailed description, the micro-nano structure surface defect intelligent detection method and system of the present invention present a comprehensive technical solution, forming a complete technical chain from image acquisition, preprocessing, deep learning analysis to model optimization and deployment. This method and system can not only efficiently and accurately detect various micro-nano structure surface defects, but also has strong adaptability and scalability, providing a powerful tool for quality control in the field of micro-nano manufacturing.

[0148] Next, the application process of the method of the present invention will be explained in detail in combination with several typical micro-nano structure samples and their common defects.

[0149] Case 1: Surface defect detection of semiconductor chips

[0150] Micro-nano structure sample: integrated circuit chip with 14nm process technology

[0151] Common defects: particle contamination, uneven etching, metal bridging, photoresist residue, etc.

[0152] Specific testing process:

[0153] 1. Image acquisition: Use the image acquisition module 1 to control the high-precision three-axis mobile platform and carry the chip sample for scanning. Use a 100x objective lens and a high-resolution CCD camera to obtain multiple 4096×3072 pixel macro images.

[0154] 2. Image stitching: stitch multiple acquired images to form an ultra-high-resolution image (e.g., 32768×24576 pixels) covering the entire chip surface.

[0155] 3. Image preprocessing: Image processing module 4 is used for preprocessing. First, a Gaussian filter (kernel size 5×5, σ=1.5) is applied to remove noise, and then the CLAHE algorithm is used for local contrast enhancement, and the image is divided into 16×16 blocks for processing.

[0156] 4. Defect detection: Image analysis module 2 uses the trained Inception-v5 structure convolutional neural network model to analyze the preprocessed image. The model input layer is set to 224×224×3, and the entire image is scanned in a sliding window manner.

[0157] 5. Result output: The system marks the various defects detected and gives an assessment of the location, type and severity. For example, a particle contamination with a diameter of about 50nm is detected at the coordinates (12500,8600), which may affect transistor performance.

[0158] 6. Performance evaluation: Use test module 3 to generate test cases that simulate different lighting conditions and scanning parameters to evaluate the detection accuracy of the system under various conditions. In this case, the system's defect detection rate under standard lighting conditions reached 98.5%, and the false alarm rate was less than 1%.

[0159] Case 2: Surface defect detection of nanoscale optical components

[0160] Micro-nano structure samples: Nanoscale precision optical lenses for laser systems

[0161] Common defects: nano-scale scratches, holes, uneven coating, etc.

[0162] Specific testing process:

[0163] 1. Image acquisition: Use the confocal microscope system equipped with the image acquisition module 1 to scan the surface of the optical lens layer by layer. Each layer scan obtains a high-definition image of 2048×2048 pixels, and the z-axis step accuracy is 5nm.

[0164] 2. 3D reconstruction: Based on the acquired multi-layer images, the image processing module 4 is used to perform 3D reconstruction to generate a high-precision 3D model of the lens surface.

[0165] 3. Feature extraction: Extract features of the 3D model, including surface roughness, local curvature, depth information, etc. These features serve as additional input channels for the convolutional neural network.

[0166] 4. Defect detection: Image analysis module 2 uses an improved 3D-Inception-v5 network model for analysis. This model adds a branch for processing 3D information based on the standard Inception-v5.

[0167] 5. Result visualization: The system generates a 3D defect distribution map of the lens surface, marking the location, size and depth information of various defects. For example, a nano-scale scratch about 100nm long and 10nm deep was detected at 3.2mm from the center of the lens.

[0168] 6. Online learning: The real-time online learning module 6 receives new test results. Especially for some new and difficult-to-classify defect types, the system will request manual confirmation and update the model to continuously improve the recognition ability of various rare defects.

[0169] Case 3: Surface defect detection of flexible electronic devices

[0170] Micro-nanostructure samples: Flexible OLED displays for wearable devices

[0171] Common defects: micro cracks, interlayer peeling, conductive material fracture, etc.

[0172] Specific testing process:

[0173] 1. Multimodal image acquisition: In addition to acquiring standard optical images, the image acquisition module 1 is also equipped with an infrared thermal imaging camera and a resistance measurement system. This allows the surface morphology, thermal distribution, and conductivity data to be acquired simultaneously.

[0174] 2. Image fusion: The image processing module 4 registers and fuses the optical image, thermal image and resistance distribution map to generate a multi-channel comprehensive information map.

[0175] 3. Dynamic deformation test: Test module 3 controls the sample holder to perform bending and stretching tests while continuously collecting images to detect defects that may occur during the deformation process.

[0176] 4. Multimodal defect detection: Image analysis module 2 uses the Inception-v5 model optimized for multimodal data for analysis. This model can comprehensively consider surface morphology, thermal distribution anomalies, and changes in conductive properties to more accurately identify various types of defects.

[0177] 5. Defect evolution prediction: Based on the results of dynamic deformation testing, the system uses a timing analysis algorithm to predict the evolution trend of defects during continuous use. For example, it is predicted that a small crack detected at coordinates (500,300) may expand to a critical size after 1,000 bends.

[0178] 6. Cloud-based collaborative optimization: The cloud-based deployment module 5 uploads the detection results and prediction models to the cloud. Data from different production lines and usage environments are aggregated and analyzed to continuously optimize the detection algorithms and prediction models.

[0179] Through these specific cases, we can see the flexibility and effectiveness of the micro-nanostructure surface defect intelligent detection method of the present invention in different application scenarios. This method can not only adapt to different types of micro-nanostructures and defects, but also integrate multimodal data to achieve dynamic detection and prediction. Through cloud deployment and online learning, the system can continuously optimize itself and adapt to new manufacturing processes and defect types, providing a comprehensive and powerful solution for quality control in the field of micro-nano manufacturing.

[0180] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, replacement, and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent detection method for surface defects of micro-nano structures, characterized in that: include: The acquisition steps include: Control the automated sample stage to move the sample in three dimensions and obtain multiple small-range macro images; splicing the multiple small-range macro images to form a high-resolution image; Processing steps include: Based on the high-resolution image, the defects on the surface of the micro-nano structure are identified by analyzing the image through a deep learning convolutional neural network model; Based on the recognition results, a test case set is generated to simulate the images obtained on the surface of micro-nano structures with different sampling parameters; Output steps include: Calculate the performance parameters of the system and evaluate the feasibility of the system; The high-resolution image is digitally processed to achieve defect detection on the surface of the micro-nano structure.

2. The intelligent detection method for micro-nanostructure surface defects according to claim 1, characterized in that: The acquisition step specifically includes: Control the three-dimensional motion platform to lift and lower vertically, and perform macro alignment between the microscope and the sample; Scan the sample through the optical system module to obtain multiple small-range images; The multiple small-range images are registered and a homography matrix is ​​calculated, a perspective transformation is performed on the homography matrix, and the multiple small-range images after the perspective transformation are spliced.

3. The intelligent detection method for micro-nanostructure surface defects according to claim 1, characterized in that: The convolutional neural network model in the processing step is an Inception-v5 structure improved based on the Inception structure, including: The first convolution module includes a 3×3 convolution layer, a 3×3 convolution layer, and a 3×3 convolution layer; Sampling module, including global pooling layer and fully connected layer; The second convolution module includes a 5×5 convolution layer, a 3×3 convolution layer, a 3×3 convolution layer, and a 3×3 convolution layer; The Inception-v5 structure module includes 1×1 convolution layer, 1×3 convolution layer, 3×1 convolution layer, 3×3 convolution layer, average pooling layer, 1×1 convolution layer, 1×3 convolution layer, 3×1 convolution layer and 3×3 convolution layer.

4. The method for intelligent detection of micro-nanostructure surface defects according to claim 3, characterized in that: The processing steps also include: Setting hyperparameters for the convolutional neural network model, including learning rate and learning rate decay; Execute the training and parameter adjustment of the convolutional neural network model, stop the training when the loss function drops to close to 0, and obtain a preliminarily trained network model; Wherein, the loss function is: Among them, L i j represents the loss function, y i represents the probability that the i-th pixel is the defect class, 1 represents the probability that the pixel is the background class, N represents the number of pixels, and i and j represent the width and length of the image respectively.

5. The method for intelligent detection of micro-nanostructure surface defects according to claim 4, characterized in that: The processing steps also include: Using a grid search method, the learning rate and the learning rate decay are adjusted to obtain multiple groups of performance parameters; Using an algorithm based on linear regression analysis, fitting an optimal learning rate and a learning rate decay according to the multiple sets of performance parameters; The optimal learning rate and the learning rate decay value are input into the initially trained network model to obtain a trained network model.

6. The method for intelligent detection of micro-nanostructure surface defects according to claim 5, characterized in that: The output step further comprises: Replace y and y′ in the loss function with y>a·y′ and input into the trained network model, where y represents the true label, i.e., whether the image pixel is a defect; y′ represents whether the system output pixel is a defect; a is a preset trainable parameter; Calculate L(y, y′) = |ya·y′| to obtain the optimized network model; The parameters of the optimized network model are set to include: a first training data set, a second training data set and a third training data set, and a training program is executed to obtain a first training model, a second training model and a third training model respectively.

7. The intelligent detection method for micro-nanostructure surface defects according to claim 6, characterized in that: The output step further comprises: Input the test data set into the first training model, the second training model and the third training model to verify the detection accuracy results of the three, and save the first training model, the second training model and the third training model with high accuracy; The test data set includes: a non-intersecting data portion of the first training data set, the second training data set, and the third training data set.

8. The intelligent detection method for micro-nanostructure surface defects according to claim 1, characterized in that: The processing step also includes the following image preprocessing steps: Using a filter to remove noise, the filter is at least one of a Gaussian filter, a median filter, a mean filter or a Kalman filter; Use edge detection algorithms to extract edge information in images to enhance the contrast of defective areas; Improve the contrast of defective areas through contrast enhancement technology, using histogram equalization or local contrast enhancement methods; Use sharpening and deblurring algorithms to improve image sharpness for better identification of tiny defects; Correct the image distortion.

9. The intelligent detection method for micro-nanostructure surface defects according to claim 1, characterized in that: It also includes a boundary suitability test step: Testing of extreme surface conditions, including extreme surface textures, nonlinearities, and irregularities; Verify the system's defect detection performance and evaluate the system's robustness and adaptability.

10. Micro-nano structure surface defect intelligent detection system, characterized in that: include: An image acquisition module is used to control the automated sample stage to move the sample carried by the stage in three dimensions, obtain multiple small-range macro images, and stitch the macro images to form a high-resolution image; An image analysis module, used to build a convolutional neural network model based on deep learning, analyze the high-resolution image, and identify defects on the surface of the micro-nano structure; A test module, used to generate a test case set, simulate images obtained on the surface of the micro-nano structure with different sampling parameters, calculate the performance parameter index of the system, and evaluate the feasibility of the system; An image processing module, used for digitally processing the high-resolution image to realize defect detection on the surface of the micro-nano structure; The convolutional neural network model in the image analysis module adopts the Inception-v5 structure improved based on the Inception structure, including a first convolution module, a sampling module, a second convolution module and an Inception-v5 structure module; The system also includes a cloud deployment module for packaging the trained model into a file and uploading it to the cloud to achieve centralized management and flexible deployment of the model; The system also includes a real-time online learning module for receiving new image data for real-time testing, and automatically updating model parameters according to the test results to achieve continuous optimization and improved adaptability of the model.

Images