A defect collaborative detection method and system based on multi-source data fusion

Abstract

The application provides a defect collaborative detection method and system based on multi-source data fusion, and relates to the technical field of defect detection. The method comprises the following steps: obtaining two-dimensional image data, three-dimensional topographic data, luminance distribution map and chrominance distribution map of a Micro-LED wafer to be detected; performing structural characterization processing on the luminance distribution map and the chrominance distribution map respectively to generate corresponding luminance feature sequence and chrominance feature sequence; the structural characterization processing is used for converting discrete distribution measurement value data into semantic feature sequence with spatial position; performing feature extraction on the two-dimensional image data and the three-dimensional topographic data to obtain corresponding two-dimensional visual feature and three-dimensional topographic feature; fusing the luminance feature sequence, the chrominance feature sequence, the two-dimensional visual feature and the three-dimensional topographic feature, and inputting into a defect detection model; obtaining a defect detection result of the Micro-LED wafer according to the output of the defect detection model.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a collaborative defect detection method and system based on multi-source data fusion. Background Technology

[0002] With the rapid development of micro-display technology, micron-sized light-emitting diodes (Micro-LEDs) are considered the core of display technology due to their advantages such as high brightness, high contrast, and long lifespan. However, display technologies represented by Micro-LED wafers extensively employ semiconductor manufacturing processes such as deposition, photolithography, etching, bonding, and packaging. In particular, full-color Micro-LED screens require massive transfer, and during the transfer process, wafers need to undergo processes such as dicing, splitting, peeling, picking, pasting, and welding. Each process can cause damage and defects to the micron-sized Micro-LED wafers, greatly limiting the manufacturing yield of Micro-LED wafers. These defects include not only physical morphology defects (such as cracks, scratches, and contaminant particles), but more critically and difficult to detect are optoelectronic functional defects (such as uneven brightness mura, color deviation, and dark spots). Optoelectronic functional defects are often closely related to material properties, epitaxial processes, and driving circuits, and may manifest as low contrast and abnormalities without obvious geometric edges, making them extremely difficult to detect by physical morphology defects alone. Therefore, achieving high-precision and high-efficiency full inspection of Micro-LED wafers is a key bottleneck in current display technology.

[0003] Currently, multi-source data fusion is commonly used to improve the comprehensiveness and robustness of detection. A typical approach involves collecting different attribute information of the target, using two-dimensional (2D) visual images to capture surface texture and patterns, and using three-dimensional (3D) topographic data to obtain height and contour information. Existing detection methods largely focus on how to more effectively align, correlate, and fuse these continuous image signals, which all belong to the visual domain. They typically rely on mature convolutional neural networks (CNNs) or visual transformers (ViTs) for feature extraction and fusion, essentially optimizing the spatial continuity and local correlation of image pixels.

[0004] However, applying this approach directly to defect detection in Micro-LED wafers, especially for detecting core functional defects, reveals fundamental problems: (1) Failure of Heterogeneous Fusion of Multi-Source Data. The functional defects of Micro-LEDs need to be quantitatively characterized by the luminance distribution map and chromaticity distribution map obtained by electrical testing. However, this type of data is essentially a discrete, gridded matrix of measurement values. Each pixel represents the luminous performance parameters of a Micro-LED chip, such as luminance value and color coordinates. It is not an optical reflection or transmission image in the traditional sense. It is fundamentally different from the continuous 2D / 3D visual images with spatial texture semantics in terms of data modality. Existing fusion models (such as CNN and ViT) are mainly used to process continuous images. If the luminance / chromaticity distribution map is simply regarded as an ordinary image channel for splicing or early fusion, it is difficult to effectively learn and understand the statistical anomaly features contained therein, such as deviation of local luminance mean and abrupt change in chromaticity variance. This leads to a serious lack of sensitivity in detecting low-contrast functional defects (such as slight Mura and progressive color shift).

[0005] (2) Imbalance between computational efficiency and model capability. Micro-LED wafers are large in size and high in resolution, resulting in a huge amount of data processing. Existing fusion models, such as the Transformer based on global self-attention, have strong global modeling capabilities, but their computational complexity increases quadratically with the sequence length, making it difficult to meet the efficiency requirements for real-time detection on the production line. On the other hand, if a lightweight model or a local attention mechanism is used, the ability to judge macroscopic defects such as global uniformity of the wafer and periodicity distribution of spot points will be lost.

[0006] (3) Rigid Feature Fusion Mechanism. Existing deep networks often employ fixed pyramid structures or residual connections for multi-scale feature fusion, with pre-defined and static information flow paths. However, Micro-LED wafers exhibit significant differences in defect types and scales: detecting nanoscale scratches requires high-resolution detail features at the lower level, while judging chip-level brightness and color anomalies requires high-level semantic features. Static fusion models cannot adaptively and dynamically combine feature information at different abstract levels based on the currently analyzed potential defect types, resulting in limited detection accuracy and poor flexibility.

[0007] In summary, there is an urgent need for a solution that can effectively integrate the deep heterogeneity between physical morphology data and optoelectronic functional data in Micro-LED wafer inspection, so as to achieve coordinated, accurate and efficient detection of physical and functional defects in Micro-LED wafers. Summary of the Invention

[0008] To address the aforementioned problems, this invention provides a defect collaborative detection method and system based on multi-source data fusion, aiming to solve the technical problem of how to effectively fuse discrete optoelectronic functional data with continuous visual feature data to achieve high-precision and high-efficiency collaborative detection of physical morphology defects and optoelectronic functional defects in Micro-LED wafers.

[0009] On the one hand, the present invention provides a defect collaborative detection method based on multi-source data fusion, the method comprising: Acquire two-dimensional image data, three-dimensional topography data, brightness distribution map, and color distribution map of the Micro-LED wafer to be inspected; The luminance distribution map and chrominance distribution map are subjected to structured characterization processing to generate corresponding luminance feature sequences and chrominance feature sequences; wherein, the structured characterization processing is used to transform discretely distributed measurement data into semantic feature sequences with spatial location; Feature extraction is performed on the two-dimensional image data and the three-dimensional shape data to obtain the corresponding two-dimensional visual features and three-dimensional shape features; The brightness feature sequence, chromaticity feature sequence, two-dimensional visual features, and three-dimensional morphology features are fused and input into the defect detection model; Based on the output of the defect detection model, the defect detection results of the Micro-LED wafer are obtained.

[0010] Furthermore, the structured representation processing is tokenized representation processing; the acquisition of the luminance feature sequence or the chrominance feature sequence specifically includes: The brightness distribution map or chromaticity distribution map is divided into multiple non-overlapping image regions; For each of the image regions, extract the statistical features of the measurements within that image region; The measured value of each image region is combined with the extracted statistical features to obtain the corresponding region feature vector; Each region feature vector is mapped to a high-dimensional semantic vector through a linear transformation layer, which serves as the token for the image region. Arrange the tokens corresponding to all image regions in the brightness distribution map or chromaticity distribution map according to their spatial positions to obtain the corresponding brightness feature sequence or chromaticity feature sequence.

[0011] Furthermore, the statistical features include one or more of the mean, variance, maximum value, minimum value, skewness, and kurtosis of the measured values ​​within the image region.

[0012] Furthermore, the defect detection model includes a heterogeneous attention mechanism module; wherein, the heterogeneous attention mechanism module is configured to employ a hybrid attention mechanism, including: The first attention unit employs a dot product attention mechanism to model the global dependencies of the luminance feature sequence and / or chrominance feature sequence. The second attention unit employs a linear attention mechanism to efficiently model the two-dimensional visual features and / or three-dimensional shape features while preserving local perception capabilities.

[0013] Furthermore, the defect detection model also includes an adaptive deep attention module; wherein, the adaptive deep attention module is configured to: during the forward propagation of the model, allow the query vector generated by the current network layer to adaptively calculate the attention weights between the key-value pairs output by the current network layer and at least one previous network layer, so as to achieve dynamic feature aggregation across network layers.

[0014] Furthermore, the training of the defect detection model specifically includes: The neural network model is pre-trained using the training dataset of Micro-LED wafers; the parameters of the pre-trained model are fine-tuned based on manually labeled data or semi-automatically labeled data to reduce the domain difference between the pre-trained data and the real data. Alternatively, the defect detection model can be obtained by directly using manually labeled data or semi-automatically labeled data to train the neural network model end-to-end.

[0015] Furthermore, manual annotation refers to manually annotating all defects on the collected training dataset; semi-automatic annotation refers to using some models to perform preliminary annotation on the collected training dataset, and then performing secondary annotation manually.

[0016] Furthermore, the defect detection model is a neural network model built on the Transformer architecture, and the first attention unit and the second attention unit are integrated in different layers or different heads of the Transformer architecture.

[0017] Furthermore, the method also includes: generating a comprehensive defect map containing physical morphological defects and optoelectronic functional defects based on the defect detection results; and classifying the Micro-LED wafer or individual chip according to the comprehensive defect map.

[0018] On the other hand, the present invention provides a defect collaborative detection system based on multi-source data fusion, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the above-described methods.

[0019] In summary, this invention provides a defect collaborative detection method and system based on multi-source data fusion, which achieves the following beneficial effects compared with existing technologies: (1) This invention, through structured characterization processing, transforms the heterogeneous data modalities of discrete, unstructured luminance / chrominance measurement matrices and continuous, textured 2D / 3D visual data in Micro-LED wafer inspection into a unified semantic space and fuses them. This processing method overcomes the limitations of traditional multimodal fusion methods such as simple channel splicing or processing only isomorphic image data, and solves the technical problem that discrete optoelectronic functional data and continuous visual feature data are difficult to deeply fuse in Micro-LED wafer inspection. This enables the defect detection model to collaboratively analyze the physical morphological defects (dependent on 2D / 3D visual features) and optoelectronic functional defects (dependent on luminance / chrominance semantic features) of the wafer. Compared with existing general defect detection methods, it significantly improves the detection sensitivity of low-contrast functional defects and realizes the comprehensive detection of physical damage and optoelectronic failure, thereby improving detection efficiency.

[0020] (2) This invention divides the luminance / chrominance distribution map into regions (Patch) and extracts their statistical features. Then, it combines the original measurements with the statistical features and performs linear projection, thereby transforming the originally discrete and isolated measurements into high-dimensional vectors rich in regional statistical semantics, thus obtaining a token sequence. This tokenization representation process forces the model to focus on regions with statistical anomalies, rather than noise in individual pixels, thereby greatly enhancing the feature extraction capability for low-contrast functional defects. At the same time, it preserves spatial location information, ensuring the accuracy of defect localization and solving the problem that traditional methods are unable to effectively learn functional defect patterns from discretely distributed measurement data.

[0021] (3) This invention achieves a dynamic balance and optimal allocation between computational efficiency and modeling accuracy by configuring a heterogeneous hybrid attention mechanism in the defect detection model, namely, dot product attention for token sequences and linear attention for visual feature maps. Specifically, this invention utilizes computationally intensive dot product attention to accurately capture long-range global dependencies in luminance / chrominance feature sequences, such as chrominance drift from wafer edge to center; at the same time, it utilizes computationally efficient linear attention to quickly process local details in high-resolution 2D / 3D visual feature maps, such as microcrack edges. This mechanism allocation, while ensuring both global functional uniformity and dual detection accuracy of local physical defects, significantly reduces the computational complexity of high-resolution wafer data processing, meeting the efficiency requirements of real-time inspection on the production line. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the method steps of a defect collaborative detection method and system based on multi-source data fusion provided by the present invention; Figure 2 This is a schematic diagram illustrating the principle of a defect collaborative detection method and system based on multi-source data fusion provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the method, step, or apparatus that includes said element.

[0026] Existing methods either fail to effectively correlate physical morphology defects with optoelectronic functional defects, or their fusion strategies are too general and struggle to capture the unique cross-modal correlation features of Micro-LEDs, such as how minute warpage leads to local color shifts. Therefore, to effectively fuse discrete optoelectronic functional data with continuous visual feature data and achieve high-precision, high-efficiency collaborative detection of physical morphology and optoelectronic functional defects in Micro-LED wafers, this invention provides a defect collaborative detection method and system based on multi-source data fusion. Through structured characterization processing, discrete luminance / chromaticity measurement data are transformed into vector sequences rich in statistical semantic features, successfully achieving deep fusion with continuous visual features within a unified semantic space. This solves the challenge of collaborative detection of physical morphology and optoelectronic functional defects generated during the manufacturing process of high-density, micron-scale Micro-LED chips.

[0027] Specifically, such as Figure 1 As shown, the method includes: S100: Acquire two-dimensional image data, three-dimensional topography data, brightness distribution map, and color distribution map of the Micro-LED wafer to be inspected.

[0028] Specifically, two-dimensional image data refers to RGB or grayscale images of the surface of a Micro-LED wafer acquired by a high-resolution camera or microscope (such as a light / dark field optical microscope), used to reflect macroscopic appearance defects of the chip, such as surface contamination, scratches, cracks, and missing electrodes.

[0029] Three-dimensional topography data refers to the three-dimensional point cloud or height matrix of a wafer surface acquired through non-contact optical profilometers (such as white light interferometers, confocal microscopes, and structured light 3D scanners). Three-dimensional topography data characterizes the physical height, step difference, coplanarity, and micro-roughness of a chip, and is used to detect three-dimensional defects such as missing bumps and uneven etching.

[0030] A luminance distribution map is a two-dimensional matrix of data collected by an imaging luminance meter under powered or light-excited conditions. Each pixel value in the luminance distribution map represents the absolute luminance value of the corresponding Micro-LED chip, and is used to identify functional defects such as dark spots and murras.

[0031] A chromaticity distribution map refers to CIE color coordinates or dominant wavelength distribution data collected by an imaging colorimeter or hyperspectral camera. It reflects the color consistency of the emitted light from a Micro-LED chip and is used to detect spectral defects such as color shift and wavelength drift.

[0032] It should be noted that 2D image data, 3D topography data, brightness distribution maps, and colorimetric distribution maps can be acquired through an integrated optical inspection platform. Specifically, 2D image data can be acquired using a high-resolution camera in bright / dark conditions; 3D topography data can be obtained through scanning with a spectral confocal sensor; brightness / colorimetric distribution maps can be simultaneously captured using an imaging luminance meter and a colorimeter in electroluminescent mode; after all data acquisition is completed, they are unified to the same coordinate system using a standard image registration algorithm.

[0033] For example, the Micro-LED wafer to be inspected is placed on a high-precision electrically controlled displacement platform. This platform employs a composite detection head integrating a high-resolution camera for acquiring two-dimensional image data, a spectral confocal sensor for acquiring three-dimensional topographic data, and a luminance / colorimeter for acquiring photoelectric data. On one hand, under bright / dark illumination, the high-resolution camera acquires two-dimensional image data of the surface texture; the spectral confocal sensor scans the wafer surface point-by-point, generating three-dimensional topographic data. On the other hand, a specific current is applied to the wafer to trigger Micro-LED emission, and the luminance and colorimeter simultaneously capture the emission images, generating luminance and chromaticity matrices. Finally, using the platform's high-precision motion control console and calibration algorithm, all data are unified to the same physical coordinate system, ensuring positional consistency for subsequent fusion analysis.

[0034] S200: Perform structured characterization processing on the luminance distribution map and chrominance distribution map respectively to generate corresponding luminance feature sequences and chrominance feature sequences. Among them, the structured characterization processing is used to transform the discretely distributed measurement data into semantic feature sequences with spatial location.

[0035] Traditional methods typically treat luminance / chrominance maps as ordinary channels and directly stitch them onto the image. However, luminance / chrominance data is inherently discrete, non-continuous pixel-level measurements, and its spatial correlation patterns differ fundamentally from those of RGB images or depth maps. To address the mismatch between discrete luminance / chrominance data and the feature patterns of continuous images / depth maps, this invention proposes structured representation processing.

[0036] As an example, the structured representation processing is tokenized representation processing; the acquisition of the luminance feature sequence or chrominance feature sequence specifically includes: Divide the luminance distribution map or chrominance distribution map into multiple non-overlapping image regions; For each image region, extract the statistical characteristics of the measurements within that region. The measured values ​​of each image region are combined with the extracted statistical features to obtain the corresponding region feature vector; Each region feature vector is mapped to a high-dimensional semantic vector through a linear transformation layer, which serves as a token for the image region. Arrange the tokens corresponding to all image regions in the luminance distribution map or chrominance distribution map according to their spatial positions to obtain the corresponding luminance feature sequence or chrominance feature sequence.

[0037] In other words, this invention divides the luminance / chrominance map into N×N local regions (Patch), extracts the statistical features of the luminance / chrominance values ​​in each region, and concatenates them with the original luminance / chrominance information. Then, through a learnable linear projection layer, each local region is mapped into a high-dimensional embedding vector. At the same time, positional encoding is added to preserve spatial information, thereby forming a semantically rich token sequence.

[0038] As an example, the statistical features include one or more of the following: mean, variance, maximum value, minimum value, skewness, and kurtosis of the measured values ​​within the image region. The measured data refers to the collected luminance or chrominance values, which are discrete and discontinuous measurements that constitute a luminance distribution map or a chrominance distribution map.

[0039] Preferably, for each image region, the mean and variance of the measurements within that region are calculated; the mean is used to characterize the average brightness or chromaticity level of the region; and the variance is used to characterize the dispersion or uniformity of the measurements within the region.

[0040] The measurement values ​​of each image region are combined with the extracted statistical features to obtain the corresponding region feature vector. Specifically, this involves: first, expanding all measurement values ​​(such as brightness or chromaticity values) within the image region in a preset order (such as row priority) to form the original measurement value vector; then, arranging the extracted statistical features (such as mean and variance) in a preset order to form the statistical feature vector; and finally, concatenating the measurement value vector and the statistical feature vector end to end on the same dimension to form a new, higher-dimensional region feature vector.

[0041] For example, if an image region contains N measurements and M statistical features are extracted, the combined region feature vector has a dimension of (N + M). This vector retains the original information and incorporates regional statistical semantic information, which facilitates unified processing in subsequent linear projection layers.

[0042] It should be noted that each region feature vector is an independent token. Each token is injected with its corresponding spatial encoding to preserve its absolute position information on the wafer. Finally, all tokens are arranged according to their spatial position within the image region, and the tokens from all image regions together form a token sequence. .

[0043] This invention utilizes structured representation processing to transform dense pixel-level data into sparse token sequences rich in semantic information. This reduces subsequent computation and forces the model to focus on regions with photoelectric anomalies rather than being overwhelmed by a large number of normal background pixels. This improves the detection sensitivity to low-contrast functional defects such as mura and color shift. It breaks through the coarse-grained processing method of traditional methods that directly stitch together luminance / chrominance distribution maps as ordinary channels, and realizes a refined representation of discrete and discontinuous photoelectric measurement data.

[0044] S300: Extract features from two-dimensional image data and three-dimensional shape data to obtain corresponding two-dimensional visual features and three-dimensional shape features.

[0045] Specifically, for two-dimensional image data, standard preprocessing operations in the field of computer vision are performed, including denoising, enhancement, and normalization, to eliminate noise interference, enhance defect features, and unify data distribution; then the original two-dimensional image data is transformed into a standardized grayscale feature matrix or RGB feature matrix, where each element of the feature matrix represents the intensity or color information of the corresponding pixel position.

[0046] For 3D topography data, filtering, registration, and meshing are performed first. Then, based on the meshing, local geometric attributes such as curvature and normal vectors are extracted at each vertex. The processed height information and extracted geometric features are then organized into a deep feature matrix suitable for deep learning. The channel dimension of this deep feature matrix includes not only height information but also the calculated curvature map and normal vector component maps, thus transforming unstructured 3D topography data into a structured feature representation that can be directly processed by convolutional neural networks.

[0047] After preprocessing the two-dimensional image data and three-dimensional shape data and constructing the structured feature matrix, the two-dimensional visual feature matrix and the three-dimensional shape feature matrix are then input into the backbone network constructed based on the convolutional neural network or the multi-scale feature extraction network for deep feature extraction, resulting in two-dimensional visual feature maps and three-dimensional shape feature maps.

[0048] It should be noted that this invention is not limited to a strict sequential processing order for multi-source heterogeneous data, but supports flexible parallel or asynchronous processing architectures to adapt to the needs of different hardware configurations and computing resources. That is to say, the structured representation of luminance / chrominance data and the feature extraction of 2D / 3D data are logically two independent processing branches, and there is no strict order dependency between them. In practical applications, these two steps can be executed in parallel or sequentially.

[0049] S400: It fuses the brightness feature sequence, chromaticity feature sequence, two-dimensional visual features and three-dimensional shape features, and inputs them into the defect detection model.

[0050] As an example, such as Figure 2 As shown, the defect detection model includes a heterogeneous attention mechanism module; wherein, the heterogeneous attention mechanism module is configured to employ a hybrid attention mechanism, including: The first attention unit employs a dot product attention mechanism to model the global dependencies of the luminance feature sequence and / or chrominance feature sequence. The second attention unit employs a linear attention mechanism to perform local detail association modeling on two-dimensional visual features and / or three-dimensional shape features.

[0051] The defect detection model of this invention is based on a hybrid attention mechanism of dot product attention (SDPA) and linear attention (LA), employing SDPA and LA at different network layers after feature concatenation. Specifically, some network layers use SDPA: by calculating a global similarity matrix on the concatenated multimodal feature sequences, a global distance dependency across modalities is established to capture complex patterns such as wafer-level brightness / chromaticity global drift and multimodal semantic associations; other layers use LA: by using kernel function approximation to reduce the computational complexity of attention, it can efficiently model high-resolution 2D / 3D visual feature maps while retaining the ability to perceive local high-frequency details such as minute scratches and particle defects. This solves the efficiency and performance contradiction between global modeling and local detail capture in a single attention mechanism, achieving an organic unity between global semantic modeling and local detail perception of multimodal features.

[0052] Existing methods typically perform attention calculations only within a single layer or simply stack multiple layers, failing to dynamically adjust cross-layer dependencies based on the input content. To address the rigidity of inter-layer information flow in traditional Transformers, this invention not only uses the key-value pairs of the current network layer for queries but also allows it to adaptively select the most relevant deep memory from all preceding network layers for aggregation.

[0053] As an example, the defect detection model also includes an adaptive deep attention module; wherein the adaptive deep attention module is configured to: during the forward propagation of the model, allow the query vector generated by the current network layer to adaptively calculate the attention weights between the key-value pairs output by the current network layer and at least one previous network layer, so as to achieve dynamic feature aggregation across network layers.

[0054] For example, for the current The layer, whose adaptive deep attention module is: ; in, , These represent the key vector and value vector of the corresponding image region in the preceding network layer, respectively. The approach is consistent with the dot product attention or linear attention mechanism used in the current network layers. Then, a defect type prediction head and a defect level prediction head are used to predict the probability of the defect type and the level of the defect, respectively. The matrix sizes of the softmax output matrices of the last layer of the defect type prediction head and the defect level prediction head are as follows: , ; , These represent the number of defect types and the number of defect levels, respectively.

[0055] This invention introduces an adaptive deep attention module into the deep layers of the network, allowing the query of the current layer to adaptively select the most relevant deep memories (key-value pairs) from all preceding layers for aggregation. This enables dynamic modeling of cross-layer dependencies, significantly improving the model's ability to model deep dependencies. It allows the model to dynamically combine features at different abstract levels based on different targets (such as defects of different sizes and types), and to more effectively utilize low-level detailed features when detecting minor defects. The current network layer query vector not only focuses on key-value pairs within its own layer but also adaptively calculates attention weights between key-value pairs output from shallower layers (containing edge details) and deeper layers (containing global semantics), thereby dynamically aggregating cross-layer information, making more accurate judgments, and improving the comprehensiveness, accuracy, and efficiency of the detection results.

[0056] As an example, the training of the defect detection model specifically includes: The neural network model is pre-trained using the training dataset of Micro-LED wafers; the parameters of the pre-trained model are fine-tuned based on manually labeled data or semi-automatically labeled data to reduce the domain difference between the pre-trained data and the real data. Alternatively, the defect detection model can be obtained by directly using manually labeled data or semi-automatically labeled data to train the neural network model end-to-end.

[0057] It should be noted that the defect detection model can be trained in two different modes to adapt to different data conditions and engineering needs.

[0058] The first approach is a two-stage training model based on pre-training with normal data and fine-tuning with real defect data. This model aims to utilize a large amount of readily available normal sample data to enable the model to prioritize learning the standard physical and optoelectronic feature distribution patterns of Micro-LED wafers. Specifically, this involves: First, pre-training the neural network model using a training dataset of real, normal Micro-LED wafers without defects. This training dataset, which does not contain artificially injected defects, includes multiple sets of collected defect-free normal 2D image data, 3D morphology data, brightness distribution maps, and chromaticity distribution maps. The core objective of this first stage is to model the baseline pattern of the wafer under normal conditions, allowing the network to fully learn its inherent 2D visual, 3D morphology, and optoelectronic signal characteristics. Subsequently, the model parameters are fine-tuned based on real defect data containing manually labeled or semi-automatically labeled defects. Through this second-stage fine-tuning process, the model can accurately distinguish and identify various abnormal defects that deviate from the normal pattern, effectively reducing the domain difference between the normal pattern distribution learned in the pre-training stage and the defect scenario, thus giving the model excellent generalization ability.

[0059] The second approach is end-to-end direct training based on real defect data. This approach is suitable for scenarios with sufficient and high-quality defect samples. In this mode, no separate pre-training step is required. Instead, the neural network model is directly trained end-to-end using real defect data, including manually or semi-automatically labeled data, until the model converges and the final defect detection model is obtained. This mode eliminates the pre-training step, making the process simpler, but it typically requires a higher quantity and diversity of defect samples in the training dataset to ensure the model can learn fully from scratch and distinguish between normal and defective features.

[0060] Furthermore, manual annotation refers to manually labeling all defects on the collected training dataset; semi-automatic annotation refers to using some models to perform preliminary annotation on the collected training dataset, and then performing secondary annotation manually.

[0061] During training, manually or semi-automatically labeled data is used as the training target. This data includes the defect types and levels of physical morphological defects and optoelectronic functional defects. Through end-to-end training, the model can automatically learn the discriminative features of defects from multidimensional data. The model output is the defect detection result for each pixel, image block, or chip unit, including defect types and levels such as normal, particle contamination, surface scratches, luminous dark spots, color cast, and Mura inhomogeneity.

[0062] As an example, the defect detection model can be a neural network model built on the Transformer architecture, where the first attention unit and the second attention unit are integrated in different layers or different heads of the Transformer architecture. By integrating specific hybrid attention mechanisms in the Transformer architecture through layering or separate heads, computational resources can be automatically allocated based on the global correlation of token sequences and the local details of feature maps in multimodal data. This optimizes global context awareness while ensuring the fine capture of local defect features, achieving an optimal balance between detection accuracy and computational efficiency.

[0063] As a specific embodiment, the defect detection model includes an input layer for multimodal data concatenation. Specifically, it first acquires preprocessed multimodal heterogeneous data: a two-dimensional visual feature matrix. 3D morphological feature matrix Brightness distribution matrix Chromaticity distribution matrix ;in, , These represent the height and width of the image, respectively. , , , These represent the matrix dimensions.

[0064] Then, the preprocessed multimodal heterogeneous data is concatenated along the channel dimension to generate a unified concatenation matrix: Among them, total characteristic latitude , The splicing allows subsequent neural network layers to simultaneously perceive the physical morphology and optoelectronic functional characteristics of the wafer within a unified tensor space.

[0065] To achieve the optimal balance between computational efficiency and feature representation capability, this invention deploys dot product attention and linear attention at different levels of the defect detection model or at different headers of the Transformer architecture, forming a hybrid attention mechanism that works in concert.

[0066] When the first attention unit and the second attention unit are integrated into different heads of the Transformer architecture, the attention head in the multi-head attention module is divided into two functionally complementary units: the first attention unit and the second attention unit.

[0067] The first attention unit assigns a specific number of attention heads to run the dot product attention mechanism. These heads focus on computing long-range dependencies on the global feature map. By calculating the global similarity matrix between the query vector Q and the key vector K, they accurately capture long-range cross-modal dependencies, ensuring the model has a deep understanding of macroscopic defect patterns such as large-area brightness inhomogeneity or overall color temperature shift on the wafer.

[0068] The second attention unit allocates the remaining attention heads to run a linear attention mechanism. When processing high-resolution visual features, such as detecting micron-level physical scratches or particle defects, these heads use kernel functions to approximate high-dimensional similarity calculations as inner product operations in the low-dimensional feature space, thereby significantly reducing computational complexity from quadratic to linear levels. This allows the model to greatly improve processing speed without sacrificing the ability to model local high-frequency details, meeting the efficiency requirements of real-time inspection on production lines.

[0069] Furthermore, besides multi-head segmentation within the same network layer, hybrid attention mechanisms can also be applied to network layers of different depths. For example, in deeper network layers, feature maps have lower spatial resolution but richer semantic information; dot-product attention is mainly deployed in these layers to fully exploit the global structural relationships contained in deep features, achieving high-precision classification of complex defect categories. In shallower network layers, feature maps retain rich spatial texture and high-frequency details; linear attention is mainly deployed in these layers to ensure optimal allocation of computational resources when processing ultra-high-resolution wafer images, avoiding memory overflow, while simultaneously enhancing the accuracy of locating underlying physical defects. For example, the ratio of SDPA to LA can be 1:3 across all layers; they can be used alternately, either regularly or irregularly. That is, SDPA is used in layer 1, LA in layers 2-4, SDPA in layer 5, and LA in layers 6-8; or SDPA in layers 1-2, and LA in layers 3-8; or SDPA in layer 1, LA in layer 2, SDPA in layer 3, and LA in layers 4-8.

[0070] As another embodiment, when the first attention unit and the second attention unit are integrated in different network layers of the Transformer architecture, dynamically allocating dot product attention and linear attention at different layers specifically includes: Linear attention is used to enhance local details. For example, the concatenation matrix is ​​first divided into non-repeating local windows, such as 16×16 pixels; then linear attention is applied within each window. Linear attention reduces computational cost through kernel function approximation, lowering the computational complexity from... Reduce to This enables it to efficiently process large-scale visual data while retaining the ability to perceive local high-frequency details (such as scratches and particles).

[0071] Hybrid attention collaboration mechanisms are used to balance global and local considerations, aiming to model global dependencies across windows, such as brightness drift from wafer edge to center, while preserving local details. For example, a channel recalibration is first performed on the output of a certain layer to enhance key modal channels, such as luminance / chrominance. Then, the attention type is dynamically selected based on feature complexity. If the feature variance is below a threshold (locally stable region), linear attention continues to be used; if the feature variance is above a threshold (globally variable region), dot product attention is switched. This adaptive attention selection mechanism achieves a balance between computational efficiency and modeling capability.

[0072] Dot product attention is used to model global semantics and capture long-range dependencies, such as defect distribution patterns across an entire wafer. For example, it can flatten the output into a sequence. Then, using multi-head dot product attention, Q, K, and V are divided into multiple heads, such as 8 heads, and the attention is computed in parallel.

[0073] Furthermore, linear attention, through kernel function mapping, transforms complex similarity calculations into linear operations in a low-dimensional space, specifically: ; in, Kernel functions, such as ReLU and other non-linear activation functions, are used to map the input query Q and key K to another feature space, avoiding the high-dimensional matrix inversion or normalization operations in traditional attention mechanisms, thus reducing computational complexity from... Reduce to .

[0074] Furthermore, dot product attention measures global dependencies between features by directly calculating the inner product between the query and the key, specifically: ; in, This represents the Softmax activation function, which operates on the attention score matrix, normalizing the original similarity scores into a probability distribution (attention weights) and ensuring that the sum of the weights at all positions is 1. Represents the key vector The feature dimension, i.e., the scaling factor.

[0075] It should be noted that, regardless of whether it is linear attention or dot product attention, the query, key, and value vectors are all obtained by performing different linear projections (fully connected layer transformations) on the original input feature matrix. , indicating the first The query vector at each position is weighted by a weight matrix. Map input features to query vectors . , indicating the first The key vector at each position, through the weight matrix Map input features to key vectors . , indicating the first The value vector at each position; through the weight matrix Map input features to value vectors . , represents the position index in the input feature sequence. This represents the total length of the sequence.

[0076] S500: Based on the output of the defect detection model, obtain the defect detection results of the Micro-LED wafer.

[0077] As an example, the method further includes: generating a comprehensive defect map containing physical morphological defects and optoelectronic functional defects based on the defect detection results, and classifying the Micro-LED wafer or individual chip according to the comprehensive defect map.

[0078] Specifically, the defect detection model outputs at least one of the following: defect type, defect location, and defect physical size. The model outputs the defect type for each location, such as cracks, murmurs, dark spots, and scratches, along with the corresponding location coordinates, physical size, and confidence score. Based on the defect detection results output by the model, a comprehensive defect distribution heatmap integrating physical morphological defects and optoelectronic functional defects is generated. Furthermore, the quality of the entire wafer or a single chip is classified according to preset quality standards, providing a comprehensive and reliable basis for subsequent repair, grading, and other process decisions.

[0079] On the other hand, the present invention provides a defect collaborative detection system based on multi-source data fusion, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the above-described methods.

[0080] Specifically, the system may include: The data acquisition and preprocessing unit is used to acquire two-dimensional image data, three-dimensional morphology data, brightness distribution map and color distribution map of the Micro-LED wafer to be inspected; The discrete data representation unit is used to perform structured representation processing on the luminance distribution map and the chrominance distribution map respectively, generating corresponding luminance feature sequences and chrominance feature sequences; wherein, the structured representation processing is used to transform the discretely distributed measurement data into semantic feature sequences with spatial location.

[0081] The visual feature extraction unit is used to extract features from two-dimensional image data and three-dimensional shape data to obtain corresponding two-dimensional visual features and three-dimensional shape features. The multi-source data fusion unit includes a defect detection model, which is used to fuse brightness feature sequences, chromaticity feature sequences, two-dimensional visual features and three-dimensional morphological features, and input them into the defect detection model; The result output unit is used to obtain the defect detection results of the Micro-LED wafer based on the output of the defect detection model.

[0082] The technical features of the system are consistent with those of the method, and will not be elaborated here.

[0083] In summary, this invention proposes a multi-source data fusion collaborative detection method based on tokenized representation, a hybrid attention mechanism, and an adaptive deep attention module. On the one hand, it transforms display characteristic data (brightness / chroma) into a structured token sequence and deeply fuses it with 2D / 3D visual data within a unified semantic space. On the other hand, it designs a hybrid attention mechanism that can efficiently handle ultra-long sequence modeling, greatly improving algorithm efficiency. In addition, an adaptive deep attention module is introduced to dynamically fuse features at different levels according to the target, thereby achieving rapid, accurate, interpretable, and energy-efficient collaborative detection of micro-nano physical defects and display non-uniformity defects on the surface of Micro-LED wafers.

[0084] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0090] Those skilled in the art will understand that all or part of the circuits in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0091] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A defect collaborative detection method based on multi-source data fusion, characterized in that, The method includes: Acquire two-dimensional image data, three-dimensional topography data, brightness distribution map, and color distribution map of the Micro-LED wafer to be inspected; The luminance distribution map and chrominance distribution map are subjected to structured representation processing to generate corresponding luminance feature sequences and chrominance feature sequences; wherein, the structured representation processing is tokenization representation processing, which is used to transform discretely distributed measurement data into semantic feature sequences with spatial location; Feature extraction is performed on the two-dimensional image data and the three-dimensional shape data to obtain the corresponding two-dimensional visual features and three-dimensional shape features; The brightness feature sequence, chromaticity feature sequence, two-dimensional visual features, and three-dimensional shape features are fused and input into the defect detection model; wherein, the defect detection model is based on a hybrid attention mechanism of dot product attention and linear attention; dot product attention is used to capture long-range global dependencies in the brightness feature sequence and / or chromaticity feature sequence, and linear attention is used to process local details in high-resolution two-dimensional visual features and / or three-dimensional shape features. Based on the output of the defect detection model, the defect detection results of the Micro-LED wafer are obtained.

2. The defect collaborative detection method based on multi-source data fusion according to claim 1, characterized in that, The acquisition of the luminance feature sequence or the chrominance feature sequence specifically includes: The brightness distribution map or chromaticity distribution map is divided into multiple non-overlapping image regions; For each of the image regions, extract the statistical features of the measurements within that region; The measured value of each image region is combined with the extracted statistical features to obtain the corresponding region feature vector; Each region feature vector is mapped to a high-dimensional semantic vector through a linear transformation layer to serve as the token of the image region; Arrange the tokens corresponding to all image regions in the brightness distribution map or chromaticity distribution map according to their spatial positions to obtain the corresponding brightness feature sequence or chromaticity feature sequence.

3. The defect collaborative detection method based on multi-source data fusion according to claim 2, characterized in that, The statistical features include one or more of the following: mean, variance, maximum value, minimum value, skewness, and kurtosis of the measured values ​​within the image region.

4. The defect collaborative detection method based on multi-source data fusion according to claim 1, characterized in that, The defect detection model includes a heterogeneous attention mechanism module; wherein, the heterogeneous attention mechanism module is configured to employ a hybrid attention mechanism, including: The first attention unit employs a dot product attention mechanism to model the global dependencies of the luminance feature sequence and / or chrominance feature sequence. The second attention unit employs a linear attention mechanism to efficiently model the two-dimensional visual features and / or three-dimensional shape features while preserving local perception capabilities.

5. A defect collaborative detection method based on multi-source data fusion according to any one of claims 1 to 4, characterized in that, The defect detection model also includes an adaptive deep attention module; wherein, the adaptive deep attention module is configured to: during the forward propagation of the model, allow the query vector generated by the current network layer to adaptively calculate the attention weights between the key-value pairs output by the current network layer and at least one previous network layer, so as to achieve dynamic feature aggregation across network layers.

6. The defect collaborative detection method based on multi-source data fusion according to claim 5, characterized in that, The training of the defect detection model specifically includes: The neural network model is pre-trained using the training dataset of Micro-LED wafers; the parameters of the pre-trained model are fine-tuned based on manually labeled data or semi-automatically labeled data to reduce the domain difference between the pre-trained data and the real data. Alternatively, the defect detection model can be obtained by directly using manually labeled data or semi-automatically labeled data to train the neural network model end-to-end.

7. The defect collaborative detection method based on multi-source data fusion according to claim 6, characterized in that, The manual annotation refers to manually annotating all defects on the collected training dataset; the semi-automatic annotation refers to using some models to perform preliminary annotation on the collected training dataset, and then performing secondary annotation manually.

8. The defect collaborative detection method based on multi-source data fusion according to claim 7, characterized in that, The defect detection model is a neural network model built on the Transformer architecture, and the first attention unit and the second attention unit are integrated in different layers or different heads of the Transformer architecture.

9. A defect collaborative detection method based on multi-source data fusion according to claim 1, characterized in that, The method further includes: generating a comprehensive defect map containing physical morphological defects and optoelectronic functional defects based on the defect detection results; and classifying the Micro-LED wafer or individual chip according to the comprehensive defect map.

10. A defect collaborative detection system based on multi-source data fusion, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images