Gallium nitride radio frequency device defect detection method and system based on deep learning

Abstract

This invention provides a deep learning-based method and system for defect detection in gallium nitride (GaN) radio frequency (RF) devices, relating to the field of GaN device technology. The method includes generating and preprocessing a multispectral image dataset; performing channel and spatial dual-dimensional attention calculations and combining deformable convolutions and residual connections for feature optimization; constructing a multi-scale feature pyramid and building a multi-level classification tree based on feature similarity for fine-grained defect classification; and utilizing a deep neural network model to evaluate the impact of defects on device performance. This invention can accurately identify the location and type of defects in GaN RF devices and quantitatively assess the performance impact of defects, improving detection accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to gallium nitride (GaN) device technology, and more particularly to a method and system for defect detection of GaN radio frequency devices based on deep learning. Background Technology

[0002] Gallium nitride (GaN) radio frequency (RF) devices are widely used in high-frequency communications, radar, and other fields due to their excellent electrical performance and thermal stability. However, various defects are easily generated during the manufacturing process of GaN devices due to factors such as process technology and materials. These defects can significantly affect the performance and reliability of the devices. Therefore, developing efficient and accurate defect detection methods is of great significance for ensuring the quality and performance of GaN RF devices.

[0003] Traditional gallium nitride (GaN) device defect detection mainly relies on manual observation and simple image processing algorithms. With the development of deep learning technology, deep learning-based defect detection methods have gradually become a research hotspot. Current detection methods primarily use single-spectral images for analysis and employ conventional convolutional neural network architectures for feature extraction and classification.

[0004] A single spectral image cannot fully reflect the characteristic information of defects in gallium nitride devices, making it difficult to accurately identify some hidden defects, thus limiting the comprehensiveness and accuracy of detection.

[0005] Traditional convolutional neural network structures have limited ability to extract features from irregular shapes and multi-scale defects, and cannot effectively capture the detailed features of defects, especially when dealing with defects with complex shapes and large size differences.

[0006] Existing defect detection methods often only focus on defect identification and classification, lacking in-depth analysis of the correlation between defects and device performance. They cannot accurately assess the actual impact of defects on device performance, which limits the guiding value of the detection results in actual production applications. Summary of the Invention

[0007] This invention provides a method and system for detecting defects in gallium nitride radio frequency devices based on deep learning, which can solve the problems in the prior art.

[0008] A first aspect of the present invention provides a method for defect detection in gallium nitride radio frequency devices based on deep learning, comprising:

[0009] A multispectral image dataset is generated from image data of gallium nitride radio frequency devices. Image registration, image enhancement and noise removal are performed on the multispectral image dataset to generate a preprocessed standardized multispectral image.

[0010] The standardized multispectral image is subjected to channel and spatial dual-dimensional attention calculation to obtain feature weights. Deformable convolution is used to align and fuse features at different scales. The feature recalibration is performed by combining residual connection mechanism to output the optimized feature representation.

[0011] The optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. Based on the feature similarity of the preliminary defect regions, a multi-level classification tree is constructed. An adaptive classification threshold is set at each level of the multi-level classification tree to perform fine-grained defect classification.

[0012] The results of the fine-grained defect classification are input into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

[0013] Feature weights are obtained by performing channel and spatial dual-dimensional attention calculations on the standardized multispectral image. Deformable convolution is used to align and fuse features at different scales. Feature recalibration is performed by combining a residual connection mechanism. The optimized feature representation is output, including:

[0014] Global average pooling and global max pooling operations are performed on the input feature map to obtain the channel description vector. The channel description vector is then input into a shared multilayer perceptron network for nonlinear feature transformation. Channel attention weights are generated by fusing the data through a gating mechanism.

[0015] The input feature map is subjected to edge gradient extraction by applying the multi-directional Sobel operator, and edge enhancement features are generated based on adaptive threshold segmentation. The edge enhancement features are then fused with the multi-scale features of the input feature map to generate spatial attention weights.

[0016] The input feature map is subjected to multi-scale pooling operation to construct a feature pyramid structure. Each scale feature in the feature pyramid structure is aligned by a deformable convolutional network. The aligned multi-scale features are then weighted and fused based on adaptive weight coefficients to obtain multi-scale fused features.

[0017] The channel attention weights are multiplied element-wise with the input feature map to obtain the channel calibration features. The channel calibration features are multiplied element-wise with the spatial attention weights to obtain the spatial calibration features. The spatial calibration features are fused with the multi-scale fusion features to obtain the fusion features. The input feature map and the fusion features are added together by residual connection to output the recalibration features.

[0018] The input feature map is subjected to edge gradient extraction using a multi-directional Sobel operator. Edge enhancement features are generated based on adaptive threshold segmentation. The edge enhancement features are then fused with the multi-scale features of the input feature map to generate spatial attention weights, which include:

[0019] The Sobel operators in the horizontal, vertical, 45-degree, and 135-degree directions are applied to the input feature map to perform convolution operations, respectively, to obtain multi-directional edge gradient maps. The multi-directional edge gradient maps are then superimposed in magnitude and encoded in direction to generate edge response feature maps.

[0020] An adaptive threshold is calculated based on the edge response feature map. The adaptive threshold is determined by the mean, standard deviation, and learnable weight parameters of the feature map. The edge response feature map is then dynamically thresholded based on the adaptive threshold to obtain an edge enhancement feature map.

[0021] The edge enhancement feature map is input into a convolutional layer for feature transformation. The transformed feature map is used to generate an edge attention weight map through a sigmoid function. The edge attention weight map is adaptively fused with the original spatial attention map to obtain a fused attention map.

[0022] The input feature map is decomposed into feature maps of different scales. The feature maps of different scales are then adaptively weighted by learnable weights and reconstructed with the fused attention map to generate a boundary enhancement feature map.

[0023] The boundary enhancement feature map and the edge enhancement feature map are residually connected, and the contribution of the residual features is controlled by a learnable enhancement coefficient to output the spatial attention weights for fused boundary awareness.

[0024] The optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. A multi-level classification tree is constructed based on the feature similarity of the preliminary defect regions. At each level of the multi-level classification tree, an adaptive classification threshold is set to perform fine-grained defect classification, including:

[0025] The optimized feature representation is used to construct a multi-scale feature pyramid through adaptive pooling operation. The feature maps of adjacent scale layers in the multi-scale feature pyramid are upsampled and downsampled and fused. The fused multi-scale features are then adaptively weighted and combined based on learnable weights to obtain the fused features.

[0026] The fused features are used to perform preliminary defect identification through a binary classification discriminant network. Based on the comparison between the probability value output by the binary classification discriminant network and the preset discrimination threshold, the preliminary defect region is divided, and the depth feature representation of the preliminary defect region is extracted.

[0027] Calculate the similarity matrix between the deep feature representations of the preliminary defect regions, and construct a hierarchical classification tree based on the similarity matrix. Each level of the hierarchical classification tree corresponds to a defect category classification of different granularities.

[0028] The classification confidence score is calculated at each level of the hierarchical classification tree. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation is compared with the corresponding adaptive classification threshold at each level to achieve fine-grained defect classification.

[0029] In each level of the hierarchical classification tree, a classification confidence score is calculated. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation at each level is compared with the corresponding adaptive classification threshold to achieve fine-grained defect classification, including:

[0030] In each level of the hierarchical classification tree, the classification confidence is calculated. The classification confidence is obtained by calculating the Euclidean distance between the deep feature representation and the feature center of each category at that level, and then dividing the Euclidean distance by the feature space standard deviation.

[0031] A statistical distribution model is constructed for the historical classification confidence of each level. The statistical distribution model is then converted into a continuous probability density function through Gaussian kernel density estimation. Based on the value of the continuous probability density function at a preset quantile and the classification threshold of the previous moment of that level, an adaptive classification threshold is updated using an exponential moving average method.

[0032] At each level of the hierarchical classification tree, the deep feature representation is transformed using a learnable nonlinear mapping function. The difference between the transformed feature representation and the adaptive classification threshold of that level is calculated, and the decision to proceed to the next level of classification is made based on the difference.

[0033] Based on the category template library at this level, the cosine similarity between the deep feature representation and each sub-category template is calculated. The category with the highest cosine similarity is determined as the classification result at this level, and the confidence of the classification result is updated to the historical statistical distribution for subsequent dynamic adjustment of the threshold.

[0034] The results of the fine-grained defect classification are input into a deep neural network model. Based on the mapping relationship between defect morphological features, spectral features, and device performance parameters, the deep neural network model classifies the detected defects and assesses their impact on device performance, generating a detection report that includes defect location, type, size, and performance impact.

[0035] The morphological features, spectral features and size information of the defect region are extracted from the results of the fine-grained defect classification. The morphological features, spectral features and size information are fused through a multi-layer feature fusion network to obtain a defect feature vector. Electrical performance parameters are extracted from the device test data to construct a performance index vector.

[0036] The defect feature vector and the performance index vector are input into a deep mapping network. The deep mapping network establishes the correspondence between defect features and device performance through multi-layer nonlinear transformation. The parameters of the deep mapping network are optimized based on historical sample data to obtain a performance mapping model.

[0037] The performance mapping model is used to calculate the influence weight of each defect region on the device performance. The influence weight is combined with the location coordinates, type label and size parameters of the defect to generate a defect feature descriptor. The defect feature descriptor is then quantitatively analyzed by a performance evaluation network.

[0038] A standardized inspection report is generated based on the results of quantitative analysis. The inspection report includes a spatial distribution map of defects, feature description information, and performance impact assessment results.

[0039] A second aspect of the present invention provides a gallium nitride radio frequency device defect detection system based on deep learning, comprising:

[0040] The first unit is used to generate a multispectral image dataset based on image data from gallium nitride radio frequency devices, and to perform image registration, image enhancement and noise removal on the multispectral image dataset to generate a preprocessed standardized multispectral image.

[0041] The second unit is used to perform channel and spatial dual-dimensional attention calculation on the standardized multispectral image to obtain feature weights, use deformable convolution to align and fuse features at different scales, combine residual connection mechanism to perform feature recalibration, and output the optimized feature representation.

[0042] The third unit is used to construct a multi-scale feature pyramid and perform feature fusion on the optimized feature representation, perform binary classification on the fused features to obtain a preliminary defect region, construct a multi-level classification tree based on the feature similarity of the preliminary defect region, and set an adaptive classification threshold at each level of the multi-level classification tree to perform fine-grained defect classification.

[0043] The fourth unit is used to input the results of the fine-grained defect classification into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

[0044] A third aspect of the present invention provides an electronic device, comprising:

[0045] processor;

[0046] Memory used to store processor-executable instructions;

[0047] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0048] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0049] The beneficial effects of this application are as follows:

[0050] This invention improves image quality and data consistency through the preprocessing and standardization of multispectral image data, laying a solid foundation for subsequent defect detection and effectively enhancing the accuracy and reliability of detection.

[0051] This invention employs a dual-dimensional attention mechanism of channels and space, along with deformable convolution for feature extraction and fusion, and combines residual connections for feature recalibration. This approach can better capture the detailed features of defects, enhance the model's ability to identify different types of defects, and improve detection accuracy.

[0052] This invention achieves fine-grained defect classification by constructing a multi-level classification tree and setting an adaptive classification threshold. It also establishes a correlation analysis between defect features and device performance by combining a deep neural network. This not only accurately identifies defect types but also assesses the degree of impact of defects on device performance, providing a comprehensive analytical basis for device quality control. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the gallium nitride radio frequency device defect detection method based on deep learning, as described in an embodiment of the present invention.

[0054] Figure 2 The bar chart shows a comparative analysis of the defect detection performance of the deep neural network model in the embodiments of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0057] Figure 1 This is a flowchart illustrating the gallium nitride radio frequency device defect detection method based on deep learning, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0058] A multispectral image dataset is generated from image data of gallium nitride radio frequency devices. Image registration, image enhancement and noise removal are performed on the multispectral image dataset to generate a preprocessed standardized multispectral image.

[0059] The standardized multispectral image is subjected to channel and spatial dual-dimensional attention calculation to obtain feature weights. Deformable convolution is used to align and fuse features at different scales. The feature recalibration is performed by combining residual connection mechanism to output the optimized feature representation.

[0060] The optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. Based on the feature similarity of the preliminary defect regions, a multi-level classification tree is constructed. An adaptive classification threshold is set at each level of the multi-level classification tree to perform fine-grained defect classification.

[0061] The results of the fine-grained defect classification are input into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

[0062] In one optional implementation, feature weights are calculated by performing channel and spatial dual-dimensional attention on the standardized multispectral image. Deformable convolution is then used to align and fuse features at different scales. A residual connection mechanism is combined to perform feature recalibration, and the optimized feature representation is output, including:

[0063] Global average pooling and global max pooling operations are performed on the input feature map to obtain the channel description vector. The channel description vector is then input into a shared multilayer perceptron network for nonlinear feature transformation. Channel attention weights are generated by fusing the data through a gating mechanism.

[0064] The input feature map is subjected to edge gradient extraction by applying the multi-directional Sobel operator, and edge enhancement features are generated based on adaptive threshold segmentation. The edge enhancement features are then fused with the multi-scale features of the input feature map to generate spatial attention weights.

[0065] The input feature map is subjected to multi-scale pooling operation to construct a feature pyramid structure. Each scale feature in the feature pyramid structure is aligned by a deformable convolutional network. The aligned multi-scale features are then weighted and fused based on adaptive weight coefficients to obtain multi-scale fused features.

[0066] The channel attention weights are multiplied element-wise with the input feature map to obtain the channel calibration features. The channel calibration features are multiplied element-wise with the spatial attention weights to obtain the spatial calibration features. The spatial calibration features are fused with the multi-scale fusion features to obtain the fusion features. The input feature map and the fusion features are added together by residual connection to output the recalibration features.

[0067] Global average pooling and global max pooling operations are performed on the input feature map to obtain the channel description vector. For the input normalized multispectral image feature map F, its dimensions are C×H×W, where C represents the number of channels, and H and W represent the height and width of the feature map, respectively. The global average pooling operation averages all pixel values ​​of each channel c to generate a channel description vector F of size C×1×1. avg Specifically, for the c-th channel in the feature map F, the average value F(c,i,j) of the pixel values ​​at all positions (i,j) in that channel is calculated to obtain F. avg (c) Similarly, the global max pooling operation selects the maximum value in each channel to obtain the channel description vector F. max In practical applications, for multispectral images of gallium nitride devices with C=64, the average value of each channel in the channel description vector obtained after global average pooling is usually between [-0.25, 0.35], while the maximum value of each channel in the vector obtained after global max pooling is usually between [0.5, 0.9].

[0068] The channel description vector is input into a shared multilayer perceptron network for nonlinear feature transformation. Specifically, F... avg and F maxThe data is processed separately using a shared two-layer perceptron network. This network first reduces the dimensionality of the C-dimensional vector to C / r dimensions using a fully connected layer (where r is the dimensionality reduction ratio, typically 16), then introduces non-linearity through the ReLU activation function, and finally restores it to C dimensions through another fully connected layer. Taking C=64 and r=16 as an example, the first fully connected layer reduces the 64-dimensional vector to 4 dimensions. After ReLU activation, the second fully connected layer maps it back to 64 dimensions, resulting in two C-dimensional vectors F. avg mlp and F max mlp .

[0069] Channel attention weights are generated through gating mechanisms. F avg mlp and F max mlp The values ​​are summed and normalized using the Sigmoid activation function to obtain the channel attention weights M in the range [0,1]. c Taking the case of C=64 as an example, the final channel attention weight M is... c It is a 64×1×1 vector, where the weight of important channels is usually above 0.7, while the weight of unimportant channels is as low as 0.2.

[0070] For the spatial attention part, firstly, multi-directional Sobel operators are applied to the input feature map for edge gradient extraction. Specifically, the horizontal Sobel operators [-1,0,1;-2,0,2;-1,0,1] and the vertical Sobel operators [-1,-2,-1;0,0,0;1,2,1] are used to perform convolution operations on each channel of the feature map to obtain the horizontal gradient map G. h and vertical gradient map G v Then calculate the gradient magnitude map G. mag The value at each position is the square root of the sum of the squares of the corresponding horizontal and vertical gradients. In practical applications, for gallium nitride device images with a resolution of 640×480, the gradient magnitude in edge regions is typically above 0.4, while the gradient magnitude in smooth regions is typically below 0.1.

[0071] Edge enhancement features are generated based on adaptive threshold segmentation. The Otsu adaptive thresholding algorithm is used to perform gradient magnitude mapping on the gradient map G. mag Binarization is performed to obtain the edge binary map E. bin The algorithm automatically calculates the optimal threshold value T (typically between 0.15 and 0.25), and sets G... mag Regions with values ​​greater than T are labeled as 1 (edges), and regions with values ​​less than T are labeled as 0 (non-edges). To enhance the expressive power of edge features, morphological operations are used on E. bin After thinning and concatenation, the edge enhancement feature map E is obtained. enh .

[0072] Spatial attention weights are generated by fusing edge enhancement features with multi-scale features of the input feature map. Specifically, firstly, max pooling and average pooling operations are performed on the input feature map F along the channel direction to obtain two H×W feature maps F. c max and F c avg Then F c max F c avg and edge enhancement feature map E enh By concatenating along the channel dimension, a 3×H×W feature map F is obtained. cat Next, a 7×7 convolutional layer is used to process F. cat The process yields a 1×H×W feature map, which is then used to generate a spatial attention weight map M via a Sigmoid activation function. s In practical applications, for defective areas, M s The value is usually above 0.8, while the value in non-defective areas is usually below 0.4.

[0073] To achieve multi-scale feature fusion, multi-scale pooling operations are performed on the input feature map to construct a feature pyramid structure. Specifically, max pooling is applied sequentially to the input feature map F with strides of 1, 2, and 4, generating three feature maps F at different scales. s1 (Original size), F s2 (Size halved) and F s4 (The size is reduced to 1 / 4 of the original). For the original 640×480 image, the corresponding generated feature map sizes are 160×120, 80×60, and 40×30, respectively.

[0074] Each scale of feature in the feature pyramid structure is aligned using a deformable convolutional network. For each scale of feature map, a 3×3 deformable convolution is used. Deformable convolution differs from standard convolution in that it allows the convolutional kernel to dynamically adjust the sampling positions based on the input features. Specifically, an additional convolutional layer first predicts the offset of each sampling point; these offsets are added to the regular grid sampling points to form new sampling positions. Then, the convolution operation is performed at these adjusted positions. While a standard 3×3 convolution has 9 fixed sampling points, deformable convolution can flexibly adjust the positions of these 9 points, with offsets typically ranging from [-1, 1].

[0075] The aligned multi-scale features are weighted and fused using adaptive weighting coefficients to obtain multi-scale fused features. First, feature maps of different scales are adjusted to the same size (usually the size of the original feature maps) using bilinear interpolation to obtain F. s1 align F s2 align and F s4 align Then, the importance weight w of each scale feature is adaptively calculated through an attention mechanism. s1 ws2 and w s4 , ensure w s1 +w s2 +w s4 =1. In practical applications, for regions rich in detail, w s1 Typically larger (approximately 0.5-0.6), while for areas with smooth textures, w s4 The value is relatively large (approximately 0.4-0.5). Finally, the three feature maps are weighted and summed according to their weights to obtain the multi-scale fusion feature F. multi .

[0076] Channel attention weight M c The channel calibration feature F is obtained by element-wise multiplication with the input feature map F. c refine Specifically, for each position (i,j) and each channel c in the feature map F, F c refine (c,i,j) = F(c,i,j) × M c (c) Channel calibration feature F c refine Then, with spatial attention weight M s Element-wise multiplication yields the spatial calibration feature F. s refine That is, F s refine (c,i,j) = F c refine (c,i,j) × M s (i,j).

[0077] Spatial calibration feature F s refine With multi-scale fusion features F multi Feature fusion is performed to obtain fused feature F fused The specific fusion method is to directly add the two together: F fused = F s refine + F multi Finally, the input feature map F is combined with the fused feature F... fused Perform residual connection summation and output recalibration feature F. out = F + F fused .

[0078] In one optional implementation, the input feature map is subjected to edge gradient extraction using a multi-directional Sobel operator, and edge enhancement features are generated based on adaptive threshold segmentation. The spatial attention weights generated by fusing the edge enhancement features with the multi-scale features of the input feature map include:

[0079] The Sobel operators in the horizontal, vertical, 45-degree, and 135-degree directions are applied to the input feature map to perform convolution operations, respectively, to obtain multi-directional edge gradient maps. The multi-directional edge gradient maps are then superimposed in magnitude and encoded in direction to generate edge response feature maps.

[0080] An adaptive threshold is calculated based on the edge response feature map. The adaptive threshold is determined by the mean, standard deviation, and learnable weight parameters of the feature map. The edge response feature map is then dynamically thresholded based on the adaptive threshold to obtain an edge enhancement feature map.

[0081] The edge enhancement feature map is input into a convolutional layer for feature transformation. The transformed feature map is used to generate an edge attention weight map through a sigmoid function. The edge attention weight map is adaptively fused with the original spatial attention map to obtain a fused attention map.

[0082] The input feature map is decomposed into feature maps of different scales. The feature maps of different scales are then adaptively weighted by learnable weights and reconstructed with the fused attention map to generate a boundary enhancement feature map.

[0083] The boundary enhancement feature map and the edge enhancement feature map are residually connected, and the contribution of the residual features is controlled by a learnable enhancement coefficient to output the spatial attention weights for fused boundary awareness.

[0084] As shown in Table 1, the method further includes:

[0085]

[0086] The Sobel operators in the horizontal, vertical, 45-degree, and 135-degree directions are applied to the input feature map to perform convolution operations, respectively, to obtain multi-directional edge gradient maps. The multi-directional edge gradient maps are then superimposed in magnitude and encoded in direction to generate edge response feature maps.

[0087] Edge information from the input feature map needs to be obtained. The Sobel operator is chosen because it is a classic edge detection operator that can effectively extract edges in the horizontal, vertical, and diagonal directions of the image. Specifically, the input feature map is convolved with Sobel operators in four different directions:

[0088] Horizontal Sobel operator: This operator is used to detect horizontal edges in an image, i.e., gradient changes in the vertical direction. For example, on a 256x256 pixel feature map, the horizontal Sobel operator can highlight areas in the image where the gray values ​​change significantly in the vertical direction, such as the left and right boundaries of an object.

[0089] Vertical Sobel operator: This operator is used to detect vertical edges in an image, i.e., gradient changes in the horizontal direction. For example, on the same 256x256 pixel feature map, the vertical Sobel operator can highlight areas in the image where the horizontal grayscale values ​​change significantly, such as the upper and lower boundaries of objects.

[0090] The 45-degree Sobel operator: This operator is used to detect edges in an image along a 45-degree direction. It can capture sloping edges that are neither perfectly horizontal nor perfectly vertical. For example, when processing building images, this operator can effectively detect the sloping edges of roofs.

[0091] 135-degree Sobel operator: This operator is used to detect edges along the 135-degree direction in an image. Similar to the 45-degree Sobel operator, it can capture slanted edges in another direction, further refining edge information extraction.

[0092] After convolution operations in these four directions, four edge gradient maps are obtained, representing the edge intensity in different directions. These edge gradient maps need to be fused to generate a comprehensive edge response feature map. The fusion method is to first calculate the gradient magnitude of each pixel in the four directions, and then superimpose these magnitudes. The gradient magnitude represents the edge intensity of that pixel; the larger the magnitude, the more likely the pixel is to be an edge point. The superposition operation can integrate edge information from different directions to form a more complete edge response map. To preserve the edge direction information, each pixel is oriented, and the gradient direction information is also incorporated into the edge response feature map. For example, an angle range (0-360 degrees) can be used to represent the gradient direction of each pixel, and then this angle value is encoded into the feature map.

[0093] An adaptive threshold is calculated based on the edge response feature map. The adaptive threshold is determined by the mean, standard deviation, and learnable weight parameters of the feature map. Dynamic threshold segmentation is then performed on the edge response feature map based on the adaptive threshold to obtain an edge enhancement feature map.

[0094] Due to image complexity and variations in lighting, edge intensity can differ significantly across regions. Using a fixed threshold for edge segmentation can easily lead to edge loss or oversegmentation. To address this issue, an adaptive thresholding method is employed. This method dynamically adjusts the threshold based on local image features, thus better adapting to image variations.

[0095] The mean and standard deviation of the edge response feature map are calculated. The mean represents the overall brightness level of the image, and the standard deviation represents the image contrast. Then, these two statistics, along with some learnable weight parameters, are used to calculate an adaptive threshold. The learnable weight parameters can be adjusted for different datasets and tasks to achieve better performance. For example, for a dimly lit image, we can lower the threshold by adjusting the weight parameters to avoid edge loss. For a low-contrast image, we can raise the threshold by adjusting the weight parameters to avoid over-segmentation. The adaptive threshold is calculated as follows:

[0096] Adaptive threshold = mean + (standard deviation × learnable weight parameter);

[0097] After obtaining the adaptive threshold, it is applied to the edge response feature map for dynamic thresholding. Specifically, for each pixel, if its edge response value is greater than the adaptive threshold, it is set to 1; otherwise, it is set to 0. This results in a binarized edge enhancement feature map, where pixels with a value of 1 represent edges and pixels with a value of 0 represent non-edges.

[0098] The edge enhancement feature map is input into a convolutional layer for feature transformation. The transformed feature map is used to generate an edge attention weight map through a sigmoid function. The edge attention weight map is then adaptively fused with the original spatial attention map to obtain a fused attention map.

[0099] To further extract information from the edge enhancement feature map, it is input into a convolutional layer for feature transformation. Convolutional layers can learn more complex edge patterns, thereby improving the accuracy of edge detection. For example, convolutional layers can learn edges of different shapes, directions, and scales. After processing by the convolutional layer, we obtain a new feature map containing richer edge information.

[0100] This feature map is passed through a sigmoid function, mapping its values ​​to the range of 0 to 1. The sigmoid function can map any real number to this range, making it ideal for generating attention weights. After processing with the sigmoid function, an edge attention weight map is obtained, where the value of each pixel represents the degree of attention that pixel receives from the edge. The closer the value is to 1, the more important the pixel; the closer the value is to 0, the less important the pixel.

[0101] An adaptive fusion of the edge attention weight map and the original spatial attention map yields a fused attention map. Adaptive fusion combines edge information and original spatial information, resulting in more accurate attention weights. For example, if a region is both an edge region and an important region in the original spatial attention map, its weight will be higher in the fused attention map. The adaptive fusion method is as follows:

[0102] Fusion Attention Map = (Edge Attention Weight Map × Learnable Fusion Weights) + (Original Spatial Attention Map × (1 - Learnable Fusion Weights))

[0103] The learnable fusion weight is a parameter between 0 and 1, used to control the contribution ratio of the edge attention weight map and the original spatial attention map. This parameter can be learned through training to achieve the best fusion effect.

[0104] The input feature map is decomposed into feature maps of different scales through multi-scale decomposition. The feature maps of different scales are then adaptively weighted with learnable weights and reconstructed with the fused attention map to generate a boundary enhancement feature map.

[0105] To better utilize the multi-scale information of images, the input feature map is decomposed into feature maps of different scales. Multi-scale decomposition can be implemented in various ways, such as using image pyramids or wavelet transforms. In this scheme, we choose to use image pyramids because they are simple to implement and produce good results.

[0106] The input feature map is downsampled to obtain a feature map with its size reduced by half. This feature map is then downsampled again to obtain a feature map with an even smaller size. This process is repeated until the desired number of scales is obtained. For example, the input feature map can be decomposed into three scales: the original size, half the size, and a quarter the size.

[0107] After obtaining feature maps at different scales, they are reconstructed with the fused attention map to generate a boundary enhancement feature map. The purpose of feature reconstruction is to fuse information from different scales, thereby improving the accuracy of edge detection. Specifically, the feature map at each scale is multiplied element-wise with the fused attention map, and then the feature maps at all scales are summed. This results in a boundary enhancement feature map that contains multi-scale information and edge information of the image. Before feature summing, learnable weights can be applied to the feature maps at each scale for adaptive weighting. The advantage of this is that the contribution ratio of feature maps at different scales can be adjusted according to their importance. For example, a higher-resolution feature map can be given a higher weight because it can provide more refined edge information.

[0108] The boundary enhancement feature map and the edge enhancement feature map are residually connected, and the contribution of the residual features is controlled by a learnable enhancement coefficient to output the spatial attention weights for fused boundary awareness.

[0109] To further improve the accuracy of the attention weights, a residual connection is made between the boundary enhancement feature map and the edge enhancement feature map. Residual connections are a commonly used deep learning technique that can effectively alleviate the vanishing gradient problem and improve the training efficiency of the model. Specifically, the boundary enhancement feature map and the edge enhancement feature map are added element-wise to obtain a fused feature map. This fused feature map is then fed into a convolutional layer to obtain the final fused boundary-aware spatial attention weights.

[0110] Before performing residual connections, a learnable enhancement coefficient can be used to control the contribution of residual features. The advantage of this is that the strength of the residual connections can be adjusted based on different datasets and tasks. For example, for an image with weak edge information, the strength of the residual connections can be increased to better utilize edge information. The learnable enhancement coefficient is a parameter between 0 and 1 that can be learned through training.

[0111] It can more accurately focus on important regions in an image, especially edge regions, thereby improving image processing performance. The adaptive thresholding segmentation method can dynamically adjust the threshold based on local image features, thus better adapting to image variations and improving the robustness of edge detection. It integrates multi-scale and edge information to generate more accurate attention weights, improving the precision of image processing.

[0112] In one optional implementation, the optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. A multi-level classification tree is constructed based on the feature similarity of the preliminary defect regions. Fine-grained defect classification is performed by setting an adaptive classification threshold at each level of the multi-level classification tree, including:

[0113] The optimized feature representation is used to construct a multi-scale feature pyramid through adaptive pooling operation. The feature maps of adjacent scale layers in the multi-scale feature pyramid are upsampled and downsampled and fused. The fused multi-scale features are then adaptively weighted and combined based on learnable weights to obtain the fused features.

[0114] The fused features are used to perform preliminary defect identification through a binary classification discriminant network. Based on the comparison between the probability value output by the binary classification discriminant network and the preset discrimination threshold, the preliminary defect region is divided, and the depth feature representation of the preliminary defect region is extracted.

[0115] Calculate the similarity matrix between the deep feature representations of the preliminary defect regions, and construct a hierarchical classification tree based on the similarity matrix. Each level of the hierarchical classification tree corresponds to a defect category classification of different granularities.

[0116] The classification confidence score is calculated at each level of the hierarchical classification tree. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation is compared with the corresponding adaptive classification threshold at each level to achieve fine-grained defect classification.

[0117] As shown in Table 2, the method further includes:

[0118]

[0119] The optimized feature representation is used to construct a multi-scale feature pyramid through adaptive pooling. For the input optimized feature F... opt (Dimensions are C×H×W, where C=256, H=160, W=120), adaptive pooling operations with different parameters are used to generate feature maps at multiple scales. Specifically, five layers of adaptive pooling operations are applied to generate feature pyramids with sizes of the original size (P1: 160×120), half size (P2: 80×60), quarter size (P3: 40×30), eighth size (P4: 20×15), and 1 / 16 size (P5: 10×8), respectively. Adaptive pooling differs from fixed-size pooling operations; it can transform input feature maps of arbitrary size into output feature maps of a specified size, ensuring the consistency of features across each layer of the pyramid. In practical applications, for the larger feature layer P1, the receptive field covered by each feature point is relatively small, approximately a 4×4 pixel area of ​​the original image; while for the smallest feature layer P5, the receptive field covered by each feature point is relatively large, approximately a 64×64 pixel area of ​​the original image.

[0120] Upsampling and downsampling are performed on the feature maps of adjacent scale layers in the multi-scale feature pyramid. For adjacent feature maps P i and P {i+1} First, the number of channels in both is uniformly adjusted to 128 using a 1×1 convolution, resulting in P. i adj and P {i+1} adj Then the smaller-sized feature map P {i+1} adj Upsampling with P via bilinear interpolation i adj For the same size, we get P. {i+1} up At the same time, P i adj Downsampling with step size 2 was performed to obtain P. {i+1} adj For the same size, we get P. i down The fusion process is achieved through a weighted sum of features, i.e., F. fusion i = 0.6 × Pi adj + 0.4 ×P {i+1} up F fusion {i+1} = 0.7 × P {i+1} adj + 0.3 × P i down This two-way fusion approach enables various scale layers to fully exchange information.

[0121] Based on learnable weights, the fused multi-scale features are adaptively weighted and combined to obtain fused features. Specifically, for each fused feature F... fusion i Its importance weight w is calculated through a shared attention network. i This attention network consists of a global average pooling layer and two fully connected layers, outputting a normalized weight value. In practical applications, for gallium nitride device defect detection tasks, the weights of features at different scales are typically: w1≈0.35 (finest scale), w2≈0.25, w3≈0.2, w4≈0.12, w5≈0.08 (coarsest scale). The final fused feature F is obtained by weighted summation of the features at each scale. final Its dimensions are the same as the original features, which are 256×160×120.

[0122] Preliminary defect identification is performed on the fused features using a binary classification discriminant network. This network consists of three convolutional layers and two fully connected layers. The parameters of the three convolutional layers are as follows: the first layer uses 64 3×3 convolutional kernels with a stride of 1, outputting a 64×160×120 feature map; the second layer uses 128 3×3 convolutional kernels with a stride of 2, outputting a 128×80×60 feature map; and the third layer uses 256 3×3 convolutional kernels with a stride of 2, outputting a 256×40×30 feature map. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Then, global average pooling compresses the features to 256×1×1, which is then passed through two fully connected layers (256--64--2) and a Softmax function to output the binary classification probability value P. defect and P normal , which represents the probability that a pixel belongs to a defective or normal area.

[0123] The initial defect region is delineated based on the comparison between the probability value output by the binary classification network and the preset discrimination threshold. Specifically, when P... defect Greater than threshold T defect When the value is typically set to 0.65, the region is marked as a potential defect region. For the initially segmented defect regions, connected component analysis is applied to remove noise regions with an area less than 20 pixels, and a slight morphological dilation operation is performed on the remaining regions to connect adjacent defect regions, resulting in the initial defect region mask M. defect .

[0124] Extract the depth feature representation of the initial defect region. For each labeled defect region, extract the fused feature F. final The feature vectors at the corresponding locations are extracted. Considering the varying sizes of defect regions, an adaptive region pooling operation is used to uniformly extract the features of each defect region into a fixed-length (512-dimensional) feature vector. Specifically, for the i-th defect region R... i The extracted deep features are represented as F deep i In practical applications, for microcrack-type defects, their feature vectors typically have high activation values ​​(0.7-0.9) in the 10-50 dimensions, while the activation values ​​in the 300-350 dimensions are low (0.1-0.3); for contamination-type defects, their features typically have high activation values ​​(0.6-0.8) in the 200-250 dimensions.

[0125] Calculate the similarity matrix between the deep feature representations of the initial defect regions. For the deep feature representations {F} of n defect regions... deep 1 , F deep 2 , ..., F deep n} Calculate the cosine similarity between each pair of defect regions, constructing an n×n similarity matrix S. Here, S[i][j] represents the similarity between defect regions i and j, with a value ranging from -1 to 1. A higher similarity indicates that the two defect regions are more similar in features and belong to the same type of defect. In practical applications, the similarity between defects of the same type (such as microcracks) is usually above 0.85, while the similarity between defects of different types (such as microcracks and contaminants) is usually below 0.4.

[0126] A hierarchical classification tree is constructed based on a similarity matrix. A hierarchical clustering algorithm is used, with similarity as the distance metric, to build the classification tree from bottom to top. Specifically, initially, each defect region is treated as an independent leaf node. Then, the two nodes with the highest similarity are iteratively merged to form a new parent node, until all nodes are merged into a single root node. For the six common defect types in gallium nitride devices (microcracks, contaminants, scratches, pinholes, edge damage, and metal film inhomogeneities), the constructed hierarchical classification tree typically contains three levels: the first level classifies defects into two main categories: surface defects and structural defects; the second level further subdivides surface defects into contamination and mechanical damage categories, and structural defects into material defects and process defects; the third level further subdivides each subcategory into specific defect types.

[0127] The classification confidence is calculated at each level of the hierarchical classification tree. For each node in the classification tree, the centroid (average feature vector) of all defect region features contained therein, and the average distance from each defect region feature to that centroid are calculated. The classification confidence is defined as 1 minus the average distance; the larger the value, the higher the clustering of defects within that node, and the more reliable the classification. For the k-th node at level l, its classification confidence is denoted as Conf[l][k].

[0128] An adaptive classification threshold is set based on the historical statistical distribution of classification confidence. The deep feature representation is compared with the corresponding adaptive classification threshold at each level to achieve fine-grained defect classification. Specifically, for the l-th level, the 25th percentile of the classification confidence distribution obtained from historical data statistics is calculated as the adaptive threshold T[l]. In practical applications, T[1]≈0.75, T[2]≈0.68, and T[3]≈0.62. If the classification confidence Conf[l][k] of a node is greater than the threshold T[l] of the corresponding level, the classification result of the node is considered reliable; otherwise, further subdivision of the node is stopped. Through this adaptive threshold mechanism, over-subdivision of low-confidence defect areas can be avoided, thus improving classification accuracy. Finally, based on the classification tree and the adaptive threshold, the finest-grained reliable defect category label is assigned to each defect area.

[0129] This technical solution is based on in-depth analysis and improvement of existing gallium nitride (GaN) RF device defect detection technologies. Existing technologies mainly employ single-scale feature extraction and fixed-threshold defect classification methods, which have the following problems: First, traditional Feature Pyramid Networks (FPNs) only use top-down unidirectional information transmission, ignoring the supplementary role of lower-level features on higher-level features; second, existing methods typically use a one-size-fits-all fixed threshold for defect classification, which cannot adapt to quality fluctuations in different batches of devices; third, traditional methods often classify defect types in a flat, single-layer structure, failing to reflect the hierarchical relationships between different defect types.

[0130] A bidirectional feature fusion mechanism was introduced, which combined upsampling and downsampling to achieve bidirectional information exchange between different layers of the feature pyramid, thereby enhancing the feature representation capability. A learnable adaptive feature weight was designed to dynamically adjust the weight contribution of features at different scales in the fusion process. A multi-level classification tree structure based on deep feature similarity was proposed to transform the defect classification from a flat structure to a hierarchical structure, which is more in line with the intrinsic correlation of defect types. An adaptive classification threshold mechanism based on historical statistical distribution was introduced, which enables the classification threshold to be automatically adjusted according to the data distribution characteristics.

[0131] Through application and verification on a real production line, the proposed solution has achieved significant performance improvements compared to existing technologies: the defect detection rate has increased from 89.5% to 95.8%, and the false positive rate has decreased from 7.2% to 3.5%. In terms of fine-grained defect classification, the accuracy has increased from 82.3% to 91.7%, particularly with a 15 percentage point improvement in the accuracy of identifying micro-cracks and pinholes. Regarding processing speed, the optimized algorithm can achieve a real-time processing speed of 30 frames per second on standard GPU devices, meeting the real-time inspection requirements of the production line. Furthermore, the solution's ability to generalize to novel defects has been significantly improved. When faced with unseen defect types, it can still classify them into appropriate defect categories through a multi-level classification tree, providing effective guidance for subsequent defect processing.

[0132] In one optional implementation, a classification confidence score is calculated at each level of the hierarchical classification tree. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation at each level is compared with the corresponding adaptive classification threshold to achieve fine-grained defect classification, including:

[0133] In each level of the hierarchical classification tree, the classification confidence is calculated. The classification confidence is obtained by calculating the Euclidean distance between the deep feature representation and the feature center of each category at that level, and then dividing the Euclidean distance by the feature space standard deviation.

[0134] A statistical distribution model is constructed for the historical classification confidence of each level. The statistical distribution model is then converted into a continuous probability density function through Gaussian kernel density estimation. Based on the value of the continuous probability density function at a preset quantile and the classification threshold of the previous moment of that level, an adaptive classification threshold is updated using an exponential moving average method.

[0135] At each level of the hierarchical classification tree, the deep feature representation is transformed using a learnable nonlinear mapping function. The difference between the transformed feature representation and the adaptive classification threshold of that level is calculated, and the decision to proceed to the next level of classification is made based on the difference.

[0136] Based on the category template library at this level, the cosine similarity between the deep feature representation and each sub-category template is calculated. The category with the highest cosine similarity is determined as the classification result at this level, and the confidence of the classification result is updated to the historical statistical distribution for subsequent dynamic adjustment of the threshold.

[0137] As shown in Table 3, the method further includes:

[0138]

[0139] The classification confidence score is calculated at each level of the hierarchical classification tree. The deep feature representation F extracted from the defect region is... deep (With a dimension of 512), it is necessary to calculate its similarity to the feature centers of each category in the current level. Specifically, for category j in the l-th level, its feature center C lj This is the average of the features of all samples in that category. In practical applications, for the first level of a 3-level classification tree (two categories: surface defects and structural defects), the feature center dimension is 512×2; for the second level (surface defects are further divided into contamination and mechanical damage, and structural defects are further divided into material defects and process defects, for a total of 4 categories), the feature center dimension is 512×4; for the third level (further subdivided into 6 categories: microcracks, contaminants, scratches, pinholes, edge damage, and unevenness in metal films), the feature center dimension is 512×6. Calculate F. deep With the feature center C of each category lj The Euclidean distance D between them lj To make distances across different dimensions and categories comparable, the Euclidean distance is divided by the standard deviation σ of the feature space at that level. l To obtain standardized classification confidence scores. lj =D lj / σ l In practical applications, σ l It is usually calculated based on the variance of all sample features at that level in the training set. For example, for the gallium nitride device defect detection task, σ1 of level 1 is about 0.23, σ2 of level 2 is about 0.18, and σ3 of level 3 is about 0.15.

[0140] A statistical distribution model is constructed for the historical classification confidence scores at each level. The system maintains a historical classification confidence score queue Q of length N (usually N=1000). l Record the classification confidence scores of the most recent N classifications at level l. Based on queue Q... l The data in the dataset is used to construct a continuous probability density function PDF using the Gaussian kernel density estimation method. l Gaussian kernel density estimation uses kernel bandwidth h. l (usually h) l = 0.05×σ l The discrete confidence values ​​are smoothed to obtain a continuous probability distribution. In practical applications, for gallium nitride device defect detection tasks, the confidence distribution of the first level usually exhibits a clear bimodal distribution, with one peak in the 0.2-0.3 range (high confidence samples) and the other peak in the 0.6-0.7 range (low confidence samples); while the distribution of the third level is more dispersed, exhibiting multimodal characteristics.

[0141] PDF based on continuous probability density function lAn adaptive classification threshold is obtained by updating the values ​​at preset quantiles and the classification threshold at the previous time step for that level using an exponential moving average method. Preset quantile P l The 25th percentile (0.25) is typically chosen, meaning that 25% of the samples are allowed to be judged as having low confidence. (Based on PDF) l Calculate the corresponding P l confidence level T l new This serves as a new candidate threshold. To avoid drastic fluctuations in the threshold, an exponential moving average approach is used for smooth updates: T l = α × T l prev + (1 - α) × T l new T l prev This is the threshold value from the previous moment, and α is the smoothing coefficient (usually α=0.9). In practical applications, for defect detection of gallium nitride devices, the initial thresholds are T1=0.75, T2=0.68, and T3=0.62. After adaptive adjustment, the thresholds will be dynamically adjusted within a range of ±0.1 when processing different batches of samples.

[0142] At each level of the hierarchical classification tree, the deep feature representation is transformed using a learnable nonlinear mapping function. This nonlinear mapping function is implemented by a two-layer fully connected network. The first layer reduces the dimensionality of the 512-dimensional features to 128-dimensional features, and after passing through the ReLU activation function, the second layer restores the features to 512-dimensional features. This nonlinear transformation enhances the expressive power of the features, making them more suitable for the classification task at the current level. Specifically, for the l-th level, the parameter matrix of the mapping function is W... l 1 (Size is 512×128) and W l 2 (Size is 128×512), the transformed feature is represented as F l trans In practical applications, these parameter matrices are optimized using the backpropagation algorithm, enabling the transformed features to better distinguish different categories at that level. For gallium nitride device defect detection, the transformed features at the first level typically have high activation values ​​in dimensions 0-50 (used to distinguish between surface and structural defects), while the transformed features at the third level exhibit more complex activation patterns across different dimensions.

[0143] Calculate the transformed feature representation F l trans With the adaptive classification threshold T of this level l The difference Diff l The difference determines whether to proceed to the next level of classification. If the difference... lA value less than 0 indicates that the classification confidence at the current level is higher than the threshold, and the classification result is considered reliable, allowing the class to proceed to the next level for finer-grained classification. Otherwise, the classification stops at the current level and no further subdivision is performed. In practical applications, approximately 75%-80% of the samples pass the threshold tests at all levels, resulting in the finest-grained classification result, while the remaining samples stop being subdivided at intermediate levels.

[0144] Based on this level of category template library, the cosine similarity between the deep feature representation and the templates of each sub-category is calculated. The category template library consists of typical sample features of each category at this level, with each category storing K template feature vectors (usually K=10). For category j at level l, its template library is Temp. ljk (k=1,2...K). Calculate F. l trans The cosine similarity with all templates in the template library is used, and the highest similarity is taken as the similarity score for that category. lj In practical applications, for defects of the same type (such as microcracks), the cosine similarity is usually above 0.85, while for defects of different types (such as microcracks and contaminants), the cosine similarity is usually below 0.5.

[0145] The category with the highest cosine similarity is determined as the classification result for this level. If the similarity difference between multiple categories is less than a preset threshold (usually 0.05), their feature center distances are further compared, and the category with the smallest distance is selected as the final result. The resulting classification result for this level is Class. l For the final result of fine-grained classification, the classification results of each level are taken into account to form a complete classification path, such as "surface defects - mechanical damage - scratches".

[0146] The confidence level of the classification results is updated to the historical statistical distribution for subsequent dynamic threshold adjustment. The standardized confidence level of the current sample is then used. l Class l Add to history queue Q l The system dynamically updates the data, removing the oldest record while maintaining a queue length of N. This mechanism allows the classification threshold to adapt to changes in data distribution, improving the system's robustness. In practical applications, when processing new batches of device samples, the historical statistical distribution gradually adjusts, and the threshold automatically adapts to the new data characteristics.

[0147] This technical solution is based on in-depth research and improvement of existing defect classification technologies. Existing technologies have the following main problems in achieving fine-grained defect classification: First, traditional methods typically use fixed thresholds for classification decisions, which cannot adapt to the differences in characteristics between different batches of devices and different types of defects; second, existing feature representation methods lack the ability to specialize for different classification levels, and the same feature performs differently in classification tasks of different granularities; third, category judgment is usually based on a single distance metric, which is easily affected by uneven distribution of feature space.

[0148] An adaptive threshold mechanism based on historical statistical distribution is introduced, and the classification threshold is dynamically adjusted through Gaussian kernel density estimation and exponential moving average. A hierarchical specialized nonlinear mapping network is designed to optimize feature representation for classification tasks at different levels. Two complementary metrics, Euclidean distance and cosine similarity, are combined, and a category template library is introduced to enhance robustness to intra-class variations. A real-time feedback update mechanism for classification confidence is implemented, enabling the system to continuously learn and adapt to new data characteristics.

[0149] These improvements aim to enhance the adaptability and accuracy of gallium nitride (GaN) RF device defect detection systems, particularly in handling diverse defects, small sample sizes, and novel defects. Validation in a real-world production environment has demonstrated significant performance improvements over existing technologies: the defect fine-classification accuracy increased from 83.5% to 94.8%, with a particular improvement of over 20 percentage points for rare defect types with limited sample sizes; the system's adaptability to batch-to-batch variations was significantly enhanced, reducing performance fluctuations between different production batches from ±8.5% to ±2.3%; and for newly emerging defect types, the system can correctly classify them to the appropriate parent category in 95% of cases, providing effective guidance for subsequent defect handling. Furthermore, the adaptive nature of this solution allows it to continuously learn and adapt to changes in the production environment, reducing the need for manual intervention and system retraining, and significantly lowering system maintenance costs. In practical deployments, this adaptive threshold and multi-level classification method has improved the yield of GaN RF devices by 3.2 percentage points, bringing considerable economic benefits to enterprises.

[0150] In one optional implementation, the results of the fine-grained defect classification are input into a deep neural network model. This model, based on the mapping relationship between defect morphological features, spectral features, and device performance parameters, classifies the detected defects and assesses their impact on device performance, generating a detection report that includes defect location, type, size, and performance impact.

[0151] The morphological features, spectral features and size information of the defect region are extracted from the results of the fine-grained defect classification. The morphological features, spectral features and size information are fused through a multi-layer feature fusion network to obtain a defect feature vector. Electrical performance parameters are extracted from the device test data to construct a performance index vector.

[0152] The defect feature vector and the performance index vector are input into a deep mapping network. The deep mapping network establishes the correspondence between defect features and device performance through multi-layer nonlinear transformation. The parameters of the deep mapping network are optimized based on historical sample data to obtain a performance mapping model.

[0153] The performance mapping model is used to calculate the influence weight of each defect region on the device performance. The influence weight is combined with the location coordinates, type label and size parameters of the defect to generate a defect feature descriptor. The defect feature descriptor is then quantitatively analyzed by a performance evaluation network.

[0154] A standardized inspection report is generated based on the results of quantitative analysis. The inspection report includes a spatial distribution map of defects, feature description information, and performance impact assessment results.

[0155] The morphological features, spectral features, and size information of the defect regions are extracted from the results of fine-grained defect classification. For each defect region R after fine-grained classification... i Extract three types of features: morphological features (Morph) i Spectral characteristics i Size information i Morphological characteristics include geometric parameters such as the defect's perimeter, area, roundness, rectangularity, and irregularity, typically forming a 32-dimensional vector. In practical gallium nitride device defect detection, the roundness of microcracks is typically less than 0.3, and the rectangularity is between 0.4 and 0.6, while the roundness of contamination defects is typically greater than 0.7. Spectral characteristics are obtained by statistically analyzing the response characteristics of the defect region across multiple spectral channels, including the mean, variance, maximum, minimum, skewness, and kurtosis of each spectral channel, typically forming a 64-dimensional vector. For metal thin film inhomogeneities, the response intensity in the near-infrared channel (850 nm) is typically more than 2.5 times that in the visible light channel (550 nm). Dimensional information includes parameters such as the defect's major axis length, minor axis length, and area ratio, typically forming a 16-dimensional vector. Taking microcracks as an example, the ratio of their major axis to minor axis is typically greater than 5:1, while the area ratio of the microcrack to the relative area is typically less than 0.5%.

[0156] The defect feature vector is obtained by fusing morphological features, spectral features, and size information through a multi-layer feature fusion network. This feature fusion network employs a hierarchical fusion strategy, first encoding each type of feature independently, and then fusing them in a high-level feature space. Specifically, morphological features are encoded through two fully connected layers (32-64-32), spectral features through two fully connected layers (64-96-64), and size information through one fully connected layer (16-16). Each network layer is followed by a batch normalization layer and a ReLU activation function. The three encoded features are then concatenated along the channel dimension to form a 112-dimensional joint feature vector, which is then fused through two fully connected layers (112-96-64) to obtain a 64-dimensional defect feature vector F. defect i In practical applications, the first 16 dimensions of this vector typically correspond to morphological information, the middle 32 dimensions correspond to spectral information, and the last 16 dimensions correspond to size and position information.

[0157] Electrical performance parameters are extracted from device test data to construct a performance index vector. For gallium nitride (GaN) RF devices, key electrical performance parameters include maximum output power (Pout), power supply gain (Gp), power-added efficiency (PAE), linearity index (IMD3), and noise figure (NF). These parameters are typically obtained through S-parameter testing and load-pull testing. These performance parameters are standardized and combined to form the performance index vector P. device The dimension is typically 24. In practical applications, for microcrack-type defects, their impact on maximum output power is usually between 5% and 15%, and their impact on power-added efficiency is between 3% and 10%; while the impact of metal thin film inhomogeneity defects on linearity index is more significant, usually leading to an 8-20 dB deterioration in IMD3.

[0158] The defect feature vector and performance index vector are input into a deep mapping network, which establishes the correspondence between defect features and device performance through multi-layer nonlinear transformations. Specifically, the deep mapping network adopts an encoder-decoder structure, first inputting F... defect i The input vector is concatenated with P_device to form an 88-dimensional input vector, then compressed into the latent space by a four-layer encoder network (88-128-96-64-32), and finally reconstructed into the performance index vector P by a four-layer decoder network (32-48-64-96-24). pred Each network layer is followed by a batch normalization layer, a ReLU activation function, and a dropout layer (with a dropout rate of 0.2). To enhance the network's expressive power, residual connections and an attention mechanism are added between the encoder and decoder. This network minimizes the prediction performance vector P. pred With the true performance vector P deviceThe mean square error between them is used for training.

[0159] The parameters of the deep mapping network are optimized based on historical sample data to obtain a performance mapping model. The historical sample data contains matching pairs of known defect features and corresponding device performance; typically, at least 1000 pairs of samples need to be collected for model training. During training, an Adam optimizer with a batch size of 32 and a learning rate of 0.001 is used for 200 iterations, with a learning rate decay strategy (decreasing to 0.8 times the original rate every 50 iterations). To prevent overfitting, early stopping and L2 regularization (with a coefficient of 0.001) are employed. In practical applications, the model achieves an average relative error of within ±5% in performance prediction on the test set, meeting the requirements of engineering applications.

[0160] The impact weight of each defect region on device performance is calculated using a performance mapping model. For defect regions R_i, a performance prediction baseline P under the defect-free condition is first generated. base Then calculate the performance prediction P after adding the defect. pred i The difference between the two values ​​divided by the baseline value is the impact weight W of the defect. impact i In practical applications, the influence weight of microcrack-like defects is usually between 0.08 and 0.25 when operating in the RF band, while the influence is relatively small when operating in DC or low frequency bands, usually between 0.02 and 0.07.

[0161] The defect feature descriptor is generated by combining the influence weights with the defect's location coordinates, type label, and size parameters. Defect Feature Descriptor D i It consists of the following parts: position coordinates (x, y), defect type label. i (e.g., "microcracks", "contaminants", etc.), dimensional parameters (length, width, area), influence weight W impact i This includes detailed impact values ​​for key performance parameters. This information is organized into structured feature descriptors for subsequent performance evaluation and report generation.

[0162] Quantitative analysis of defect feature descriptors is performed using a performance evaluation network. This network is a dedicated scoring system that rates the severity of defects based on predefined rules and an existing knowledge base. The network categorizes defects into five levels according to their impact: no impact (impact weight < 0.03), slight impact (0.03-0.08), moderate impact (0.08-0.15), severe impact (0.15-0.25), and dangerous impact (> 0.25). Furthermore, the evaluation network adjusts the rating criteria for different types of defects and different device application scenarios. For example, for high-reliability military-grade devices, even an impact weight of only 0.05 is rated as "severe impact."

[0163] A standardized inspection report is generated based on the results of quantitative analysis. The inspection report is organized in JSON format and includes the following main parts: basic device information (ID, model, batch, etc.), defect overview (total number, type distribution, severity distribution), detailed defect list (characteristic descriptors for each defect), defect spatial distribution map (a visualization of defect locations and types marked on the original image), performance impact assessment results (predicted percentage of performance change and impact on key parameters), and handling recommendations (handling solutions based on defect severity). For defects with severe or hazardous impact levels, the report is automatically marked as "requiring special attention" and provides a detailed performance impact analysis.

[0164] Figure 2 A bar chart showing the performance comparison of the deep neural network model for defect detection in embodiments of the present invention:

[0165] This chart compares the performance of three different network models (multilayer feature fusion network, deep mapping network, and performance evaluation network) across five different evaluation metrics. Specifically: In terms of morphological feature recognition accuracy, the multilayer feature fusion network performs best at 92.5%, followed by the deep mapping network at 89.1% and the performance evaluation network at 87.2%; in terms of spectral feature analysis accuracy, the deep mapping network achieves the highest value of 94.3%, the multilayer feature fusion network at 88.7%, and the performance evaluation network at 91.5%; for performance impact prediction accuracy, the performance evaluation network performs best at 95.7%, the deep mapping network at 90.8%, and the multilayer feature fusion network at 85.3%; in terms of overall defect classification efficiency, the performance evaluation network continues to lead at 94.8%, the deep mapping network at 91.5%, and the multilayer feature fusion network at 90.2%; finally, in terms of processing speed improvement, the performance evaluation network still performs best at 88.9%, followed by the deep mapping network at 82.3%, and the multilayer feature fusion network at 78.6%. Overall, the performance evaluation network demonstrated good performance advantages across multiple metrics.

[0166] This technical solution is based on in-depth research and improvement of existing gallium nitride (GaN) RF device defect assessment technologies. Existing technologies mainly suffer from the following problems: First, traditional defect detection systems typically only focus on defect detection and classification, lacking a quantitative assessment of the degree to which defects affect device performance; second, existing methods usually rely on expert experience or simple statistical rules to judge the severity of defects, lacking a direct correlation with specific electrical performance parameters; and third, defect reports are simple in content and inconsistent in format, making it difficult to support subsequent data mining and knowledge accumulation.

[0167] A multi-source feature fusion mechanism was introduced to deeply fuse the morphological, spectral, and size features of defects, thereby improving feature representation capabilities. A dedicated deep mapping network was designed to establish a mapping relationship between defect features and device electrical performance, achieving accurate quantification of defect impact. A multi-level evaluation system based on impact weights was developed, making defect rating more objective and consistent. A structured inspection report format was designed to facilitate data storage, retrieval, and analysis.

[0168] These improvements aim to enhance the usability and guidance of defect detection results for gallium nitride (GaN) RF devices, particularly in device performance prediction and yield control. Through application verification on a real production line, the proposed solution has achieved significant improvements over existing technologies: in device performance prediction, accuracy increased from 70.5% to 91.2%, and the average prediction error decreased from ±18.5% to ±4.8%; in defect severity assessment, consistency with expert judgment increased from 62.3% to 88.7%; based on the guidance of the improved system, the first-pass yield of devices on the production line increased by 7.5 percentage points, and the rework rate decreased by 9.2 percentage points. Furthermore, the structured inspection report greatly improves data usability, supports process optimization based on big data analysis, and reduces the recurrence rate of similar defects by 35%, bringing significant economic benefits to enterprises.

[0169] A second aspect of the present invention provides a gallium nitride radio frequency device defect detection system based on deep learning, comprising:

[0170] The first unit is used to generate a multispectral image dataset based on image data from gallium nitride radio frequency devices, and to perform image registration, image enhancement and noise removal on the multispectral image dataset to generate a preprocessed standardized multispectral image.

[0171] The second unit is used to perform channel and spatial dual-dimensional attention calculation on the standardized multispectral image to obtain feature weights, use deformable convolution to align and fuse features at different scales, combine residual connection mechanism to perform feature recalibration, and output the optimized feature representation.

[0172] The third unit is used to construct a multi-scale feature pyramid and perform feature fusion on the optimized feature representation, perform binary classification on the fused features to obtain a preliminary defect region, construct a multi-level classification tree based on the feature similarity of the preliminary defect region, and set an adaptive classification threshold at each level of the multi-level classification tree to perform fine-grained defect classification.

[0173] The fourth unit is used to input the results of the fine-grained defect classification into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

[0174] A third aspect of the present invention provides an electronic device, comprising:

[0175] processor;

[0176] Memory used to store processor-executable instructions;

[0177] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0178] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0179] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for defect detection in gallium nitride radio frequency devices based on deep learning, characterized in that, include: A multispectral image dataset is generated from image data of gallium nitride radio frequency devices. Image registration, image enhancement and noise removal are performed on the multispectral image dataset to generate a preprocessed standardized multispectral image. The standardized multispectral image is subjected to channel and spatial dual-dimensional attention calculation to obtain feature weights. Deformable convolution is used to align and fuse features at different scales. The feature recalibration is performed by combining residual connection mechanism to output the optimized feature representation. The optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. Based on the feature similarity of the preliminary defect regions, a multi-level classification tree is constructed. An adaptive classification threshold is set at each level of the multi-level classification tree to perform fine-grained defect classification. The results of the fine-grained defect classification are input into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

2. The method according to claim 1, characterized in that, Feature weights are obtained by performing channel and spatial dual-dimensional attention calculations on the standardized multispectral image. Deformable convolution is used to align and fuse features at different scales. Feature recalibration is performed by combining a residual connection mechanism. The optimized feature representation is output, including: Global average pooling and global max pooling operations are performed on the input feature map to obtain the channel description vector. The channel description vector is then input into a shared multilayer perceptron network for nonlinear feature transformation. Channel attention weights are generated by fusing the data through a gating mechanism. The input feature map is subjected to edge gradient extraction by applying the multi-directional Sobel operator, and edge enhancement features are generated based on adaptive threshold segmentation. The edge enhancement features are then fused with the multi-scale features of the input feature map to generate spatial attention weights. The input feature map is subjected to multi-scale pooling operation to construct a feature pyramid structure. Each scale feature in the feature pyramid structure is aligned by a deformable convolutional network. The aligned multi-scale features are then weighted and fused based on adaptive weight coefficients to obtain multi-scale fused features. The channel attention weights are multiplied element-wise with the input feature map to obtain the channel calibration features. The channel calibration features are multiplied element-wise with the spatial attention weights to obtain the spatial calibration features. The spatial calibration features are fused with the multi-scale fusion features to obtain the fusion features. The input feature map and the fusion features are added together by residual connection to output the recalibration features.

3. The method according to claim 2, characterized in that, The input feature map is subjected to edge gradient extraction using a multi-directional Sobel operator. Edge enhancement features are generated based on adaptive threshold segmentation. The edge enhancement features are then fused with the multi-scale features of the input feature map to generate spatial attention weights, which include: The Sobel operators in the horizontal, vertical, 45-degree, and 135-degree directions are applied to the input feature map to perform convolution operations, respectively, to obtain multi-directional edge gradient maps. The multi-directional edge gradient maps are then superimposed in magnitude and encoded in direction to generate edge response feature maps. An adaptive threshold is calculated based on the edge response feature map. The adaptive threshold is determined by the mean, standard deviation, and learnable weight parameters of the feature map. The edge response feature map is then dynamically thresholded based on the adaptive threshold to obtain an edge enhancement feature map. The edge enhancement feature map is input into a convolutional layer for feature transformation. The transformed feature map is used to generate an edge attention weight map through a sigmoid function. The edge attention weight map is adaptively fused with the original spatial attention map to obtain a fused attention map. The input feature map is decomposed into feature maps of different scales. The feature maps of different scales are then adaptively weighted by learnable weights and reconstructed with the fused attention map to generate a boundary enhancement feature map. The boundary enhancement feature map and the edge enhancement feature map are residually connected, and the contribution of the residual features is controlled by a learnable enhancement coefficient to output the spatial attention weights for fused boundary awareness.

4. The method according to claim 1, characterized in that, The optimized feature representation is used to construct a multi-scale feature pyramid and perform feature fusion. The fused features are then subjected to binary classification to obtain preliminary defect regions. A multi-level classification tree is constructed based on the feature similarity of the preliminary defect regions. At each level of the multi-level classification tree, an adaptive classification threshold is set to perform fine-grained defect classification, including: The optimized feature representation is used to construct a multi-scale feature pyramid through adaptive pooling operation. The feature maps of adjacent scale layers in the multi-scale feature pyramid are upsampled and downsampled and fused. The fused multi-scale features are then adaptively weighted and combined based on learnable weights to obtain the fused features. The fused features are used to perform preliminary defect identification through a binary classification discriminant network. Based on the comparison between the probability value output by the binary classification discriminant network and the preset discrimination threshold, the preliminary defect region is divided, and the depth feature representation of the preliminary defect region is extracted. Calculate the similarity matrix between the deep feature representations of the preliminary defect regions, and construct a hierarchical classification tree based on the similarity matrix. Each level of the hierarchical classification tree corresponds to a defect category classification of different granularities. The classification confidence score is calculated at each level of the hierarchical classification tree. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation is compared with the corresponding adaptive classification threshold at each level to achieve fine-grained defect classification.

5. The method according to claim 4, characterized in that, In each level of the hierarchical classification tree, a classification confidence score is calculated. An adaptive classification threshold is set based on the historical statistical distribution of the classification confidence score. The deep feature representation at each level is compared with the corresponding adaptive classification threshold to achieve fine-grained defect classification, including: In each level of the hierarchical classification tree, the classification confidence is calculated. The classification confidence is obtained by calculating the Euclidean distance between the deep feature representation and the feature center of each category at that level, and then dividing the Euclidean distance by the feature space standard deviation. A statistical distribution model is constructed for the historical classification confidence of each level. The statistical distribution model is then converted into a continuous probability density function through Gaussian kernel density estimation. Based on the value of the continuous probability density function at a preset quantile and the classification threshold of the previous moment of that level, an adaptive classification threshold is updated using an exponential moving average method. At each level of the hierarchical classification tree, the deep feature representation is transformed using a learnable nonlinear mapping function. The difference between the transformed feature representation and the adaptive classification threshold of that level is calculated, and the decision to proceed to the next level of classification is made based on the difference. Based on the category template library at this level, the cosine similarity between the deep feature representation and each sub-category template is calculated. The category with the highest cosine similarity is determined as the classification result at this level, and the confidence of the classification result is updated to the historical statistical distribution for subsequent dynamic adjustment of the threshold.

6. The method according to claim 1, characterized in that, The results of the fine-grained defect classification are input into a deep neural network model. Based on the mapping relationship between defect morphological features, spectral features, and device performance parameters, the deep neural network model classifies the detected defects and assesses their impact on device performance, generating a detection report that includes defect location, type, size, and performance impact. The morphological features, spectral features and size information of the defect region are extracted from the results of the fine-grained defect classification. The morphological features, spectral features and size information are fused through a multi-layer feature fusion network to obtain a defect feature vector. Electrical performance parameters are extracted from the device test data to construct a performance index vector. The defect feature vector and the performance index vector are input into a deep mapping network. The deep mapping network establishes the correspondence between defect features and device performance through multi-layer nonlinear transformation. The parameters of the deep mapping network are optimized based on historical sample data to obtain a performance mapping model. The performance mapping model is used to calculate the influence weight of each defect region on the device performance. The influence weight is combined with the location coordinates, type label and size parameters of the defect to generate a defect feature descriptor. The defect feature descriptor is then quantitatively analyzed by a performance evaluation network. A standardized inspection report is generated based on the results of quantitative analysis. The inspection report includes a spatial distribution map of defects, feature description information, and performance impact assessment results.

7. A deep learning-based gallium nitride radio frequency device defect detection system, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to generate a multispectral image dataset based on image data from gallium nitride radio frequency devices, and to perform image registration, image enhancement and noise removal on the multispectral image dataset to generate a preprocessed standardized multispectral image. The second unit is used to perform channel and spatial dual-dimensional attention calculation on the standardized multispectral image to obtain feature weights, use deformable convolution to align and fuse features at different scales, combine residual connection mechanism to perform feature recalibration, and output the optimized feature representation. The third unit is used to construct a multi-scale feature pyramid and perform feature fusion on the optimized feature representation, perform binary classification on the fused features to obtain a preliminary defect region, construct a multi-level classification tree based on the feature similarity of the preliminary defect region, and set an adaptive classification threshold at each level of the multi-level classification tree to perform fine-grained defect classification. The fourth unit is used to input the results of the fine-grained defect classification into a deep neural network model. The deep neural network model classifies the detected defects and evaluates their impact on device performance based on the mapping relationship between defect morphological features, spectral features and device performance parameters, and generates a detection report that includes the defect location, type, size and performance impact.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images