Method for automatic defect detection classification of sapphire substrates based on visual inspection

Abstract

The application discloses a method for automatic defect detection and classification of sapphire substrates based on visual detection, relates to the technical field of visual detection and image processing, and comprises the following steps: performing adaptive threshold segmentation and phase consistency detection on a multi-modal image dataset, fusing the adaptive threshold segmentation and the phase consistency detection, and obtaining a joint defect probability graph; extracting a defect candidate region from the joint defect probability graph and generating a unified feature representation; inputting the unified feature representation into a double-branch neural network, fusing and classifying the unified feature representation through a cross-attention mechanism, outputting a defect classification result, reconstructing the defect classification result by using a NeRF algorithm, outputting a three-dimensional morphology of the defect, and generating a three-dimensional morphology atlas; generating a defect heat map based on the three-dimensional morphology atlas, performing space-time correlation analysis in combination with equipment process parameters, and outputting a sapphire substrate defect process analysis report. The application improves image contrast and defect characterization capability, thereby significantly enhancing the comprehensiveness and sensitivity of detection.

Description

Technical Field

[0001] This invention relates to the field of visual inspection and image processing technology, and in particular to a method for automatic defect detection and classification of sapphire substrates based on visual inspection. Background Technology

[0002] Sapphire substrates, as a key basic material for high-end electronic and optoelectronic devices, are widely used in LED manufacturing, semiconductor devices, and optical components due to their excellent mechanical strength, thermal stability, and superior optical properties. With the electronics industry's ever-increasing quality requirements for sapphire substrates, defect detection technology research has gradually become a core element in ensuring device performance and yield. In recent years, visual inspection technology has rapidly developed in sapphire substrate defect identification. Traditional visual inspection methods mainly rely on a single imaging mode, such as optical microscopy or laser scanning imaging, combined with image processing algorithms to identify and classify defects. Furthermore, the development of multimodal imaging technology in recent years, combining the advantages of multiple imaging modes, has effectively improved the accuracy and robustness of defect detection.

[0003] While existing technologies have achieved some success in sapphire substrate defect detection, several limitations remain. Current single-imaging modes struggle to comprehensively capture the complex morphological information of defects, making it difficult to accurately identify some minute or complex defects. Traditional image processing methods are unstable under varying lighting conditions and surface reflections, affecting the accuracy of defect localization. Furthermore, there is limited support for 3D defect morphology reconstruction and spatiotemporal correlation analysis with process parameters, hindering the provision of effective data for subsequent process optimization. Achieving efficient fusion of multimodal image data and accurate defect identification, combined with 3D morphology reconstruction and process parameter correlation analysis, is a key issue in improving the level of sapphire substrate defect detection. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for automatic defect detection and classification of sapphire substrates based on visual inspection, which solves the problems of incomplete defect identification and insufficient correlation analysis between defect morphology and process parameters in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for automatic defect detection and classification of sapphire substrates based on visual inspection, which includes preprocessing and optically calibrating the sapphire substrate, and outputting calibration parameters and a reference image.

[0008] Using calibration parameters and reference images, the metasurface lens is controlled to switch between confocal mode and dynamic polarization mode, and a multimodal image dataset of sapphire substrate is acquired in conjunction with a checkerboard scanning path.

[0009] Adaptive threshold segmentation and phase consistency detection are performed on a multimodal image dataset, and the data are then fused to obtain a joint defect probability map.

[0010] Defect candidate regions are extracted from the joint defect probability map, and a unified feature representation is generated;

[0011] The unified feature representation is input into a two-branch neural network, fused and classified through a cross-attention mechanism, and the defect classification result is output. The NeRF algorithm is used to reconstruct the defect and output the three-dimensional morphology, and generate a three-dimensional morphology map.

[0012] Defect thermal maps are generated based on three-dimensional topography maps, and spatiotemporal correlation analysis is performed in conjunction with equipment process parameters to output a sapphire substrate defect process analysis report.

[0013] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates according to the present invention, the specific steps for preprocessing and optically calibrating the sapphire substrate, and outputting calibration parameters and a reference image are as follows.

[0014] The sapphire substrate was fixed using a high-precision vacuum stage, and the surface was cleaned using an ion air knife.

[0015] After the sapphire substrate has been cleaned, it is automatically focused by dynamically adjusting the Z-axis position, while simultaneously positioning the crystal orientation mark and establishing a reference coordinate system.

[0016] Whiteboard flat-field correction is performed on the imaging of the metasurface lens based on the reference coordinate system to obtain pixel-level gain coefficients, and the phase calibration accuracy of the metasurface lens is verified. The calibration parameters and reference image are then output.

[0017] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the method involves controlling the metasurface lens to switch between confocal and dynamic polarization modes using calibration parameters and a reference image, and acquiring a multimodal image dataset of the sapphire substrate using a checkerboard scanning path. The specific steps are as follows:

[0018] Sub-pixel coordinates are extracted from the reference image using calibration parameters, and the phase modulation parameters of the metasurface lens are adjusted by combining the Z-axis focus offset and the plane correction matrix in the calibration parameters.

[0019] Based on the reference coordinate system, a checkerboard scanning path is planned, and the metasurface lens is switched between confocal mode and dynamic polarization mode in milliseconds through the stepping motion of the closed-loop piezoelectric stage.

[0020] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the steps of performing adaptive threshold segmentation and phase consistency detection on the multimodal image dataset, and then fusing them to obtain a joint defect probability map, are as follows:

[0021] Pattern label parsing and classification extraction are performed on multimodal image datasets to obtain surface structure images in confocal mode and polarization phase images in dynamic polarization mode;

[0022] Flat-field correction and dynamic range compression are performed on the surface structure image. Defect masks are generated from the preprocessed surface structure image using Sauvola adaptive threshold segmentation. At the same time, phase consistency features of the polarization phase image are extracted through a multi-scale Gabor filter bank.

[0023] Geometric shape, surface texture and polarization response are extracted from the defect mask, input into a lightweight neural network to generate initial fusion weights, and the defect mask and phase consistency features are preliminarily weighted and fused to obtain local region features;

[0024] The initial fusion weights are further analyzed and optimized using local region features. The optimized fusion weights are then used to perform secondary fusion of the defect mask and phase consistency features, and a joint defect probability map is output.

[0025] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the specific steps for extracting candidate defect regions from the joint defect probability map and generating a unified feature representation are as follows:

[0026] Neighborhood connectivity analysis is performed on the joint defect probability map to filter out defect candidate regions. Each defect candidate region is cropped to a fixed size and then subjected to bilinear interpolation scaling. After morphological optimization, a unified feature representation is output.

[0027] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the method involves: inputting a unified feature representation into a dual-branch neural network, fusing and classifying it through a cross-attention mechanism, outputting the defect classification result, reconstructing it using the NeRF algorithm, outputting the three-dimensional morphology of the defect, and generating a three-dimensional morphology map. The specific steps are as follows:

[0028] The unified feature representation is input into the two-branch neural network, the spatial structure features are output using the confocal branch, and the higher-order frequency domain representation is output through the polarization branch.

[0029] The spatial structural features and high-order frequency domain representations are fused into a fusion decision vector through a cross-attention mechanism. The vector is then passed through three fully connected layers for nonlinear transformation and dimensionality compression. Finally, the defect type and confidence level are output by an MLP classifier.

[0030] Using defect type and confidence level, the NeRF algorithm is used to reconstruct the three-dimensional morphology of the joint defect probability map under the constraint of sapphire refractive index, outputting the three-dimensional morphology of the defects. The three-dimensional morphology of the defects is then uniformized by the coordinate system, and the spatial coordinates and corresponding classification labels of all defect points are extracted to generate a three-dimensional morphology map.

[0031] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the method of using confocal branches to output spatial structure features refers to sequentially performing convolution extraction, nonlinear activation, and normalization processing on the pre-local texture descriptor with unified feature representation through a three-layer convolutional neural network, gradually mining the spatial structure information of the joint defect probability map, and outputting spatial structure features.

[0032] High-order frequency domain representation output via polarization branch refers to using a 4-head Transformer structure combined with multi-head attention mechanism and position encoding to model the long-distance dependencies between post-polarization phase descriptors with unified feature representation, and output a high-order frequency domain representation.

[0033] As a preferred embodiment of the visual inspection-based automatic defect detection and classification method for sapphire substrates described in this invention, the steps of generating a defect heat map based on a three-dimensional topography map, performing spatiotemporal correlation analysis in conjunction with equipment process parameters, and outputting a sapphire substrate defect process analysis report are as follows.

[0034] The coordinate system of the three-dimensional topography map is normalized to extract the spatial coordinates and corresponding classification labels of all defect points, and the Kriging interpolation algorithm is used to generate a defect heat map.

[0035] By aligning the defect heatmap with the equipment process parameters through spatiotemporal coordinates, a defect-process spatiotemporal map structure is constructed using the ST-GCN network. In the spatial dimension, the defect heatmap is matched with the distribution of equipment process parameters, and in the temporal dimension, the process batch sequence is traced, outputting a sapphire substrate defect process analysis report.

[0036] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in the first aspect of the present invention.

[0037] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in the first aspect of the present invention.

[0038] The beneficial effects of this invention are as follows: by controlling the switching between confocal and dynamic polarization modes of the metasurface lens, and in conjunction with the acquisition of multimodal image datasets through a checkerboard scanning path, the simultaneous perception of multiple physical properties of the sapphire substrate surface is achieved, improving image contrast and defect characterization capabilities, thereby significantly enhancing the comprehensiveness and sensitivity of the detection. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a method for automatic defect detection and classification of sapphire substrates based on visual inspection.

[0041] Figure 2 A flowchart for sapphire substrate pretreatment and optical calibration.

[0042] Figure 3 The flowchart shows the preprocessing of a multimodal image dataset and the generation of a joint defect probability map.

[0043] Figure 4 This is a flowchart for defect classification and 3D morphology reconstruction using a two-branch neural network. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a method for automatic defect detection and classification of sapphire substrates based on visual inspection, comprising the following steps:

[0048] S1. Perform pretreatment and optical calibration on the sapphire substrate, and output calibration parameters and reference images.

[0049] S1.1. The sapphire substrate is fixed by a high-precision vacuum stage and the surface is cleaned by an ion air knife.

[0050] It should be noted that a high-precision vacuum stage is used to fix the sapphire substrate, ensuring its stability and positional accuracy during the inspection process. An ion air knife is used for surface cleaning, employing an ionized gas stream to purge the sapphire substrate surface, removing adhering microparticles and electrostatically adsorbed impurities, thus ensuring the cleanliness of the sapphire substrate surface before image acquisition.

[0051] S1.2. The sapphire substrate with cleaned surface is automatically focused by dynamically adjusting the Z-axis position, and crystal orientation marks are positioned to establish a reference coordinate system.

[0052] It should be noted that the sapphire substrate surface images are acquired layer by layer along the Z-axis, and the Laplacian gradient variance value of each sapphire substrate surface image is calculated. The position corresponding to the sapphire substrate surface image with the largest Laplacian gradient variance value is selected as the optimal focal plane under the current field of view, thereby ensuring the longitudinal positioning accuracy of the imaging system. At the same time, based on the completion of focusing, the scale-invariant feature transformation algorithm is used to extract the local key points of the inherent crystal orientation mark region on the sapphire substrate surface and their corresponding feature descriptors, and match them with the features in the preset standard crystal orientation mark feature database. The random sampling consensus (RANSAC) algorithm is used to eliminate mismatched point pairs, retaining the high-confidence matching point set. Based on these matching points, the crystal orientation angle and spatial pose of the sapphire substrate are calculated, and a unified reference coordinate system is further established for geometric correction and position alignment in the subsequent image acquisition and processing process.

[0053] It should also be noted that by acquiring images of multiple batches of high-quality sapphire substrate samples, extracting representative structural feature points and grayscale texture information in their crystal orientation regions, and performing normalization processing and feature encoding, a standard crystal orientation marker feature database was finally constructed.

[0054] S1.3. Perform whiteboard flat field correction on the imaging of the metasurface lens based on the reference coordinate system, obtain pixel-level gain coefficients, verify the phase calibration accuracy of the metasurface lens, and output calibration parameters and reference image.

[0055] It should be noted that, after completing autofocus and crystal orientation positioning, a standard white board is placed on the plane of the sapphire substrate, and multiple images of the white board under uniform illumination are acquired under the same lighting conditions. Statistical analysis of the pixel grayscale values ​​of the acquired white board images is performed, and the gain coefficient of each pixel relative to the global average response value is calculated to form a flat-field correction matrix, which is used to compensate for image brightness inconsistencies caused by non-uniformity of optical components or differences in sensor response. Simultaneously, for phase calibration, a reference phase pattern with known design parameters is loaded onto the metasurface lens, and its corresponding point spread function image in confocal mode is acquired. Frequency domain information is extracted using Fourier transform and compared with the theoretical phase response to evaluate whether the actual phase modulation accuracy meets the preset tolerance requirements. The final output includes calibration parameters including pixel-level gain coefficients and phase calibration errors, as well as a reference image under standard imaging conditions.

[0056] S2. Using calibration parameters and reference images, control the metasurface lens to switch between confocal mode and dynamic polarization mode, and collect multimodal image datasets of sapphire substrates in conjunction with a checkerboard scanning path.

[0057] S2.1 Extract sub-pixel coordinates from the reference image using calibration parameters, and adjust the phase modulation parameters of the metasurface lens by combining the Z-axis focus offset and the plane correction matrix in the calibration parameters.

[0058] It should be noted that, using the pixel-level gain coefficient and phase calibration error information in the calibration parameters, sub-pixel-level feature point localization is performed on the reference image. Specifically, interpolation operations are used to identify the precise coordinate positions of crystal orientation marks or structural features in the reference image, obtaining sub-pixel-level coordinate data relative to the whole-pixel grid. Combined with the pre-calibrated Z-axis focusing offset in the calibration parameters, compensation is made for the longitudinal position changes required by the metasurface lens when switching between different imaging modes, ensuring that the focal plane remains consistent during the switching between confocal mode and dynamic polarization mode.

[0059] It should also be noted that the Z-axis focus offset in the calibration parameters is obtained by acquiring images of the sapphire substrate surface layer by layer in the Z-axis direction, calculating the Laplacian gradient variance of each layer of image, selecting the image position with the largest Laplacian gradient variance as the optimal focus position, and comparing it with the set initial focus position.

[0060] The flat field correction matrix in the calibration parameters is formed by acquiring multiple whiteboard images under standard lighting conditions, statistically analyzing the grayscale response value of each pixel, calculating its gain coefficient relative to the global average grayscale, and creating a pixel-level gain coefficient matrix for brightness non-uniformity compensation.

[0061] S2.2. Based on the reference coordinate system, plan the checkerboard scanning path and use the stepping motion of the closed-loop piezoelectric stage to switch the metasurface lens between confocal mode and dynamic polarization mode in milliseconds.

[0062] It should be noted that, based on the previously established reference coordinate system, a regular checkerboard scanning trajectory is divided on the sapphire substrate surface to ensure complete coverage of the image acquisition area without redundancy. A closed-loop piezoelectric stage is precisely moved along the X and Y axes at set step sizes to achieve row-by-row, column-by-column positioning control of the sapphire substrate. After each movement to the correct position, the metasurface lens is controlled to switch between confocal and dynamic polarization modes at millisecond intervals. The specific switching timing is synchronously controlled by a pixel-level gain coefficient matrix to match the optical response characteristics of different imaging modes. Images of the corresponding area are acquired in each mode, sequentially completing the multimodal image acquisition task of surface structure information in confocal mode and polarization phase information in dynamic polarization mode, ultimately outputting a multimodal image dataset.

[0063] S3. Perform adaptive threshold segmentation and phase consistency detection on the multimodal image dataset, and then fuse them to obtain a joint defect probability map.

[0064] S3.1 Perform pattern label parsing and classification extraction on the multimodal image dataset to obtain surface structure images in confocal mode and polarization phase images in dynamic polarization mode.

[0065] It should be noted that metadata parsing is performed on each multimodal image to identify the corresponding imaging mode type. Specifically, this is done by reading the system configuration parameters or file header information recorded during image acquisition to determine whether the image belongs to confocal imaging mode or dynamic polarization imaging mode. Images belonging to confocal mode are classified as surface structure images to reflect the morphological characteristics of the sapphire substrate surface; images belonging to dynamic polarization mode are classified as polarization phase images to characterize the polarization response characteristics of the material surface. The surface structure image in confocal mode and the polarization phase image in dynamic polarization mode are output respectively.

[0066] S3.2. Flat field correction and dynamic range compression are performed on the surface structure image. Defect mask is generated from the preprocessed surface structure image using Sauvola adaptive threshold segmentation. At the same time, phase consistency features of polarization phase image are extracted through multi-scale Gabor filter bank.

[0067] It should be noted that the pixel-level gain coefficient matrix obtained in the previous step is used to compensate the surface structure image pixel by pixel to eliminate brightness differences caused by uneven response. Subsequently, the compensated surface structure image undergoes dynamic range compression processing, and the contrast of the region of interest is expanded through a nonlinear mapping method to enhance the distinction between defects and the background. After the above preprocessing, the Sauvola adaptive thresholding method is used to binarize the surface structure image. The local adaptive threshold of each pixel is dynamically calculated based on the mean and standard deviation within the local window to generate a defect mask for identifying potential defect regions. For the polarization phase image, a multi-scale Gabor filter bank is applied for feature extraction. Specifically, Gabor kernel functions with multiple direction and frequency parameters are used to convolve the polarization phase image to obtain the texture direction response at different scales, and the phase consistency features of each pixel are further calculated.

[0068] S3.3 Extract the geometry, surface texture and polarization response from the defect mask, input them into a lightweight neural network to generate initial fusion weights, and perform preliminary weighted fusion of the defect mask and phase consistency features to obtain local region features.

[0069] It should be noted that geometric features, including spatial structure information such as the area, perimeter, and shape factor of the defect region, are extracted from the defect mask; surface texture features, including local gray-level distribution statistics and edge orientation histograms, are extracted; and polarization response features, based on the intensity change trend of the defect region under dynamic polarization modes, are extracted. These three types of features are combined to form an input feature vector, which is then fed into a lightweight neural network for processing. This lightweight neural network consists of a multi-layer fully connected network and uses the ReLU activation function for nonlinear mapping, outputting initial fusion weights corresponding to the defect mask and phase consistency features. The initial fusion weights are then used to perform a pixel-by-pixel weighted summation of the defect mask and phase consistency features to generate local region features with preliminary semantic fusion characteristics.

[0070] It should also be noted that,

[0071] S3.4. Further analyze and optimize the initial fusion weights using local region features, and use the optimized fusion weights to perform secondary fusion of the defect mask and phase consistency features to output a joint defect probability map.

[0072] It should be noted that, based on local region features, including the spatial structure information, surface texture features, and polarization response features of the defect region, the correlation and consistency of each feature in spatial distribution are calculated. Feature response regions with high confidence under different modalities are identified, and statistical analysis of multi-dimensional features such as pixel value distribution, edge gradient intensity, texture consistency, and spatial location is performed to extract their mean, variance, distribution pattern, and other statistical properties. Based on the statistical properties of the local region features, the initial fusion weights are dynamically adjusted. During the dynamic adjustment process, a sliding window strategy is used to normalize the weight distribution within the local neighborhood, enhancing the adaptability and discriminative ability of the fusion weights among different defect types. The optimized fusion weights are applied to a secondary weighted fusion between the defect mask and phase consistency features, calculating the weighted sum pixel by pixel, and finally outputting a joint defect probability map.

[0073] S4. Extract candidate defect regions from the joint defect probability map and generate a unified feature representation.

[0074] S4.1 Perform neighborhood connectivity analysis on the joint defect probability map to select defect candidate regions. Clip each defect candidate region to a fixed size and perform bilinear interpolation scaling. After morphological optimization, output a unified feature representation.

[0075] It should be noted that the joint defect probability map is processed using multi-scale connected component analysis and unsupervised clustering methods. Based on the confidence level and spatial distribution characteristics of the clusters, low-confidence regions are automatically eliminated, and potential defect candidate regions are selected. For each identified defect candidate region, the region is cropped according to its bounding box, and the cropped joint defect probability map is adjusted to a fixed size, for example, by scaling it to 128x128 pixels using bilinear interpolation to ensure size consistency. Subsequently, morphological operations such as opening and closing operations are applied to optimize the cropped joint defect probability map, eliminating small holes and noise, smoothing edge features, and extracting feature vectors from the optimized defect candidate regions. The output is then truncated before the last fully connected layer using a convolutional neural network model to obtain a unified feature representation.

[0076] It should also be noted that the images of potential defect candidate regions and their corresponding labels selected by multi-scale connected component analysis and unsupervised clustering are used as training data and input into the convolutional neural network model. After multiple rounds of forward and backward propagation, the network parameters are optimized to improve the convolutional neural network model's ability to extract features of defect regions, thus obtaining the trained convolutional neural network model.

[0077] S5. Input the unified feature representation into the dual-branch neural network, fuse and classify it through the cross-attention mechanism, output the defect classification result, reconstruct it using the NeRF algorithm, output the three-dimensional shape of the defect, and generate a three-dimensional shape map.

[0078] S5.1 Input the unified feature representation into the two-branch neural network respectively, use the confocal branch to output the spatial structure features, and output the frequency domain high-order representation through the polarization branch.

[0079] It should be explained that the specific method of using the confocal branch to output spatial structure features is to sequentially perform convolution extraction, nonlinear activation and normalization processing on the pre-local texture descriptor of the unified feature representation through a three-layer convolutional neural network, and gradually mine the spatial structure information of the joint defect probability map to output spatial structure features.

[0080] The high-order frequency domain representation output by polarization branching is specifically achieved by using a 4-head Transformer structure combined with multi-head attention mechanism and position encoding to model the long-distance dependency between post-polarization phase descriptors with unified feature representation and output high-order frequency domain representation.

[0081] It should also be noted that training the dual-branch neural network specifically involves: taking the historical unified feature representation as input and feeding it into the confocal branch and polarization branch respectively; using labeled defect samples to train the dual-branch neural network end-to-end; minimizing the joint loss function of spatial structured features and frequency domain features; and using the backpropagation algorithm to iteratively optimize the parameters of the convolutional layer and Transformer module, so that the dual-branch neural network can accurately extract high-dimensional defect features in the spatial and frequency domains, thereby improving the robustness and accuracy of defect identification.

[0082] S5.2. Spatial structural features and high-order frequency domain representations are fused into a fusion decision vector through a cross-attention mechanism, and then nonlinear transformation and dimensionality compression are performed through three fully connected layers. Finally, the defect type and confidence level are output by an MLP classifier.

[0083] It should be noted that spatial structured features and frequency domain features are used as the sources of queries, keys, and values, respectively. A correlation response relationship between the two is established along the channel dimension. By calculating the similarity score between feature vectors, an attention weight matrix is ​​generated, enabling spatial structured features to dynamically focus on key response regions in the frequency domain features. Simultaneously, the frequency domain features can also guide the enhancement regions of the spatial features, thereby achieving semantic alignment and complementary fusion between the two features to generate a fusion decision vector. Subsequently, the fusion decision vector is sequentially input into three fully connected layers. Each layer performs linear transformations and nonlinear activation operations, progressively mapping features from high to low dimensions, and finally compressing them to a feature dimension suitable for the classification task. The feature vector output from the fully connected layers is input into a multilayer perceptron classifier. The final layer outputs classification scores corresponding to scratches, particle contamination, and lattice distortion defects on the sapphire substrate surface. After normalization using the Softmax function, the confidence level for each defect type is obtained.

[0084] It should also be noted that the defect categories include typical defect types on the sapphire substrate surface such as scratches, particle contamination, and lattice distortion. These types were automatically extracted from samples with significant differences in geometric and polarization responses in historical image datasets through clustering and pattern analysis.

[0085] S5.3. Using the defect type and confidence level, the NeRF algorithm is used to reconstruct the three-dimensional morphology of the joint defect probability map under the constraint of sapphire refractive index, outputting the three-dimensional morphology of the defect. The three-dimensional morphology of the defect is then uniformized by the coordinate system, and the spatial coordinates and corresponding classification labels of all defect points are extracted to generate a three-dimensional morphology map.

[0086] It should be noted that the defect type and its corresponding confidence level information are used as prior constraints input into the NeRF algorithm. During the 3D spatial reconstruction process, the known refractive index parameters of the sapphire substrate material are used to perform multi-view geometric reasoning and optical path inversion on the 2D defect region corresponding to the joint defect probability map, reconstructing the morphological structure of each defect point in 3D space. Through coordinate system unification, the spatial coordinates of all defect points are transformed to a unified reference coordinate system, and the defect type label corresponding to each defect point is extracted based on the classification results. Finally, the spatial location information and classification attribute information of the defects are integrated to generate a 3D morphological map with spatial distribution characteristics.

[0087] S6. Generate a defect thermal map based on the three-dimensional topography map, perform spatiotemporal correlation analysis in combination with equipment process parameters, and output a sapphire substrate defect process analysis report.

[0088] S6.1 Perform coordinate system unification processing on the three-dimensional topography map, extract the spatial coordinates and corresponding classification labels of all defect points, and use the Kriging interpolation algorithm to generate a defect heat map.

[0089] It should be noted that the spatial coordinates of each defect point are transformed from the local imaging coordinate system during the original acquisition to a globally unified reference coordinate system to ensure that all defect points are comparable within the same spatial reference frame. Based on this, the three-dimensional spatial coordinates of each defect point and its corresponding defect type label are extracted to form a structured defect distribution dataset. Subsequently, the Kriging interpolation algorithm is used to continuously model the spatial distribution of defect points. Based on the known location and classification information of defect points, the defect density estimate at any location on the entire sapphire substrate surface is deduced, thereby generating a defect heatmap.

[0090] S6.2. By aligning the spatiotemporal coordinates, the defect heat map is mapped and associated with the equipment process parameters. The ST-GCN network is used to construct the defect-process spatiotemporal map structure. In the spatial dimension, the defect heat map and the distribution of equipment process parameters are matched. In the temporal dimension, the process batch sequence is traced, and a sapphire substrate defect process analysis report is output.

[0091] It should be noted that the spatial coordinates of the defect heatmap are matched with the equipment process parameters to ensure an accurate spatial correspondence. Then, based on the time series of process batches, the defect heatmap and equipment process parameters are traced over time to form a complete spatiotemporal coordinate mapping relationship. A spatiotemporal graph convolutional network is used to construct a spatiotemporal graph structure of sapphire substrate defects and equipment process parameters. The spatiotemporal graph convolutional network captures the spatial correlation characteristics of defects and equipment process parameters in the spatial dimension and analyzes the dynamic evolution of the process batch sequence in the temporal dimension, ultimately outputting a sapphire substrate defect process analysis report.

[0092] It should also be noted that the training of the ST-GCN network specifically involves using historical defect heatmaps and corresponding spatiotemporal sequences of equipment process parameters as training samples to supervise the training of the spatiotemporal graph convolutional network (ST-GCN network). Labeled defect category or defect severity information is used to minimize prediction error. The network parameters are iteratively optimized through the backpropagation algorithm, enabling the spatiotemporal graph convolutional network (ST-GCN network) to accurately capture the spatiotemporal correlation features between defects and process parameters, thereby improving the accuracy and reliability of defect process analysis.

[0093] The equipment process parameters are derived from real-time monitoring data and historical records of each process step in the sapphire substrate manufacturing process, including but not limited to key process indicators such as temperature, pressure, atmosphere composition, deposition rate, and doping concentration.

[0094] This embodiment also provides a computer device applicable to the method of automatic defect detection and classification of sapphire substrates based on visual inspection, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method of automatic defect detection and classification of sapphire substrates based on visual inspection as proposed in the above embodiment.

[0095] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0096] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the method for automatic defect detection and classification of sapphire substrates based on visual inspection as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0097] In summary, this invention achieves simultaneous perception of multiple physical properties of sapphire substrate surfaces by controlling the switching between confocal and dynamic polarization modes of metasurface lenses and acquiring multimodal image datasets in conjunction with a checkerboard scanning path. This improves image contrast and defect characterization capabilities, solves the problem of difficulty in capturing subtle defects under traditional single illumination methods, and thus significantly enhances the comprehensiveness and sensitivity of detection.

[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for automatic defect detection and classification of sapphire substrates based on visual inspection, characterized in that: include, Perform pretreatment and optical calibration on the sapphire substrate, and output calibration parameters and reference images; Using calibration parameters and reference images, the metasurface lens is controlled to switch between confocal mode and dynamic polarization mode, and a multimodal image dataset of sapphire substrate is acquired in conjunction with a checkerboard scanning path. Adaptive threshold segmentation and phase consistency detection are performed on a multimodal image dataset, and the data are then fused to obtain a joint defect probability map. Defect candidate regions are extracted from the joint defect probability map, and a unified feature representation is generated; The unified feature representation is input into a two-branch neural network, fused and classified through a cross-attention mechanism, and the defect classification result is output. The NeRF algorithm is used to reconstruct the defect and output the three-dimensional morphology, and generate a three-dimensional morphology map. Defect thermal maps are generated based on three-dimensional topography maps, and spatiotemporal correlation analysis is performed in conjunction with equipment process parameters to output a sapphire substrate defect process analysis report.

2. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 1, characterized in that: The preprocessing and optical calibration of the sapphire substrate, outputting calibration parameters and a reference image, are performed in the following specific steps. The sapphire substrate was fixed using a high-precision vacuum stage, and the surface was cleaned using an ion air knife. After the sapphire substrate has been cleaned, it is automatically focused by dynamically adjusting the Z-axis position, while simultaneously positioning the crystal orientation mark and establishing a reference coordinate system. Whiteboard flat-field correction is performed on the imaging of the metasurface lens based on the reference coordinate system to obtain pixel-level gain coefficients, and the phase calibration accuracy of the metasurface lens is verified. The calibration parameters and reference image are then output.

3. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 2, characterized in that: The process involves using calibration parameters and a reference image to control the metasurface lens to switch between confocal and dynamic polarization modes, and acquiring a multimodal image dataset of the sapphire substrate using a checkerboard scanning path. The specific steps are as follows: Sub-pixel coordinates are extracted from the reference image using calibration parameters, and the phase modulation parameters of the metasurface lens are adjusted by combining the Z-axis focus offset and the plane correction matrix in the calibration parameters. Based on the reference coordinate system, a checkerboard scanning path is planned, and the metasurface lens is switched between confocal mode and dynamic polarization mode in milliseconds through the stepping motion of the closed-loop piezoelectric stage. By using pixel-level gain coefficients to synchronously control the switching sequence of confocal mode and dynamic polarization mode, multimodal image acquisition is performed on sapphire substrates, and a multimodal image dataset is output.

4. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 3, characterized in that: The steps for adaptive threshold segmentation and phase consistency detection of the multimodal image dataset, followed by fusion to obtain a joint defect probability map, are as follows: Pattern label parsing and classification extraction are performed on multimodal image datasets to obtain surface structure images in confocal mode and polarization phase images in dynamic polarization mode; Flat-field correction and dynamic range compression are performed on the surface structure image. Defect masks are generated from the preprocessed surface structure image using Sauvola adaptive threshold segmentation. At the same time, phase consistency features of the polarization phase image are extracted through a multi-scale Gabor filter bank. Geometric shape, surface texture and polarization response are extracted from the defect mask, input into a lightweight neural network to generate initial fusion weights, and the defect mask and phase consistency features are preliminarily weighted and fused to obtain local region features; The initial fusion weights are further analyzed and optimized using local region features. The optimized fusion weights are then used to perform secondary fusion of the defect mask and phase consistency features, and a joint defect probability map is output.

5. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 4, characterized in that: The specific steps for extracting candidate defect regions from the joint defect probability map and generating a unified feature representation are as follows: Neighborhood connectivity analysis is performed on the joint defect probability map to filter out defect candidate regions. Each defect candidate region is cropped to a fixed size and then subjected to bilinear interpolation scaling. After morphological optimization, a unified feature representation is output.

6. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 5, characterized in that: The process involves inputting a unified feature representation into a dual-branch neural network, fusing and classifying the data through a cross-attention mechanism, outputting the defect classification result, reconstructing the defect using the NeRF algorithm, outputting the three-dimensional morphology of the defect, and generating a three-dimensional morphology map. The specific steps are as follows: The unified feature representation is input into the two-branch neural network, the spatial structure features are output using the confocal branch, and the higher-order frequency domain representation is output through the polarization branch. The spatial structural features and high-order frequency domain representations are fused into a fusion decision vector through a cross-attention mechanism. The vector is then passed through three fully connected layers for nonlinear transformation and dimensionality compression. Finally, the defect type and confidence level are output by an MLP classifier. Using defect type and confidence level, the NeRF algorithm is used to reconstruct the three-dimensional morphology of the joint defect probability map under the constraint of sapphire refractive index, outputting the three-dimensional morphology of the defects. The three-dimensional morphology of the defects is then uniformized by the coordinate system, and the spatial coordinates and corresponding classification labels of all defect points are extracted to generate a three-dimensional morphology map.

7. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 6, characterized in that: The aforementioned use of confocal branch to output spatial structure features refers to sequentially performing convolution extraction, nonlinear activation, and normalization on the pre-local texture descriptor with unified feature representation through a three-layer convolutional neural network, gradually mining the spatial structure information of the joint defect probability map, and outputting spatial structure features. The aforementioned output of higher-order frequency domain representations via polarization branch refers to using a 4-head Transformer structure combined with multi-head attention mechanism and position encoding to model the long-distance dependencies between post-polarization phase descriptors with unified feature representations, and output higher-order frequency domain representations.

8. The method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in claim 6, characterized in that: The process involves generating a defect thermal map based on a 3D topography map, performing spatiotemporal correlation analysis based on equipment process parameters, and outputting a sapphire substrate defect process analysis report. The specific steps are as follows: The Kriging interpolation algorithm is used to perform spatial continuity modeling and distribution estimation on the three-dimensional topography map to generate a defect heat map; By aligning the defect heatmap with the equipment process parameters through spatiotemporal coordinates, a defect-process spatiotemporal map structure is constructed using the ST-GCN network. In the spatial dimension, the defect heatmap is matched with the distribution of equipment process parameters, and in the temporal dimension, the process batch sequence is traced, outputting a sapphire substrate defect process analysis report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the method for automatic defect detection and classification of sapphire substrates based on visual inspection as described in any one of claims 1 to 8.

Images