Microelectronic device surface defect detection method based on improved YOLOv9

Abstract

The invention relates to a microelectronic device surface defect detection method based on improved YOLOv9, and the method comprises the steps: collecting a microelectronic device surface image, forming a microelectronic device surface image sample set after defect classification and marking, carrying out data enhancement to obtain an expanded data set, and dividing the expanded data set into a training set, a verification set and a test set; an efficient multi-scale attention mechanism module is introduced into a backbone network backbone of the YOLOv9 model, and a microelectronic device surface defect detection model based on the improved YOLOv9 is constructed; training the microelectronic device surface defect detection model based on the improved YOLOv9 by using the training samples in the training set to obtain an optimized microelectronic device surface defect detection model; and obtaining a to-be-detected microelectronic device image, inputting the to-be-detected microelectronic device image into the optimized microelectronic device surface defect detection model for detection, and outputting a detection classification result. The problem that the defect detection effect under small samples and small targets is poor can be solved, and the accuracy of microelectronic device surface defect detection is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic component surface defect detection, and in particular relates to a microelectronic device surface defect detection method based on improved YOLOv9. Background Art

[0002] Microelectronic devices are the core of modern electronic technology, and their surface quality has a crucial impact on the performance and reliability of the devices. With the continuous miniaturization and high integration of microelectronic devices, the size and type of surface defects are also changing. This requires surface defect detection technology to continuously improve its accuracy and sensitivity to meet new detection needs.

[0003] In the early stages of the development of surface defect detection, the simplest and most intuitive method was to rely on visual inspection by the naked eye to observe whether there were defects on the surface of the object. However, this method was too dependent on human subjective judgment and might be affected by external factors such as visual fatigue and lack of concentration, leading to problems such as missed detection and false detection. In addition, the speed and efficiency of manual visual inspection were relatively low, making it difficult to meet the needs of large-scale and high-efficiency production.

[0004] With the advancement of technology and the increase in automation needs, manual visual inspection has gradually been replaced by automated inspection methods such as machine vision. The traditional surface defect detection method based on machine vision is to detect defects on the surface of the product through image processing and analysis technology, which has improved the detection speed and degree of automation to a certain extent. However, this method requires a stable lighting environment and high-quality image acquisition equipment to ensure the quality and clarity of the image. When the ambient light is insufficient or the image acquisition quality is poor, it may cause image blur, noise and other problems, affecting the accuracy of the detection. On the other hand, the feature extraction process of this method is more dependent on manually designed features. Once the defect type changes slightly, the detection accuracy of this method drops sharply, and it is difficult to adapt to complex and changeable defect types.

[0005] With the rapid development of deep learning technology, researchers have begun to try to use deep learning methods to detect surface defects of microelectronic devices. The surface defect detection method based on deep learning uses a deep learning network model to automatically extract features and identify defects from images. This method can automatically learn to extract features from images without manually designing feature extractors, thereby avoiding the influence of human factors on the detection results. On the other hand, the deep learning model can learn the general representation of features through a large amount of training data, so it has a strong generalization ability and can identify unseen defect types. However, in industrial production, the number of surface anomalies of microelectronic devices is limited, resulting in a lack of sufficient data for training deep learning network models, namely the small sample problem. On the other hand, the surface anomalies of microelectronic devices are usually small in size, and most of them are targets with fewer pixels, namely the small target problem. Small samples and small targets will lead to poor accuracy and poor detection effect of surface defect detection of microelectronic devices. Therefore, there is an urgent need for a high-precision, high-efficiency, and strong generalization method for surface defect detection of microelectronic devices to solve the challenges of small samples and small targets in this field. Summary of the invention

[0006] In order to improve the model feature extraction capability while reducing the computational overhead, solve the problem of poor defect detection effect under small samples and small targets, and achieve more efficient, more accurate and more generalized microelectronic device surface defect detection, the present invention proposes a microelectronic device surface defect detection method based on improved YOLOv9, which includes:

[0007] Step 1: Collect surface images of microelectronic devices with and without surface defects, classify and annotate the defects respectively, and form a sample set of surface images of microelectronic devices;

[0008] Step 2: Perform data enhancement on the microelectronic device surface image sample set to obtain an expanded data set, and divide the expanded data set into a training set, a validation set, and a test set;

[0009] Step 3: Introduce an efficient multi-scale attention mechanism module into the backbone network of the YOLOv9 model to build a microelectronic device surface defect detection model based on the improved YOLOv9;

[0010] Step 4: Use the training samples in the training set to train the microelectronic device surface defect detection model based on the improved YOLOv9 to obtain an optimized microelectronic device surface defect detection model;

[0011] Step 5: Obtain the image of the microelectronic device to be inspected, input it into the optimized microelectronic device surface defect detection model for detection, and output the detection classification result.

[0012] Furthermore, in step 3, the construction process of the microelectronic device surface defect detection model based on the improved YOLOv9 is specifically as follows:

[0013] The second output end of the RepNCSPELAN4 module in the 9th layer in the YOLOv9 backbone network is connected to the input end of the EMA module, and the output end of the EMA module is connected to the input end of the ELAN-SPP module in the network Head.

[0014] Beneficial effects of the present invention: First, the present invention proposes a method for detecting surface defects of microelectronic devices based on improved YOLOv9. The method uses the latest YOLOv9 as the basic framework, introduces an efficient multi-scale attention mechanism EMA module, effectively reduces the interference of background factors, increases the algorithm's attention to abnormal target areas, and improves the model's ability to detect abnormal small targets. In addition, while improving the model's feature extraction capabilities, the attention mechanism effectively reduces the computational overhead, and uses multi-scale parallel subnets to establish short-term and long-term dependencies. The parallel structure saves time overhead while shortening the network depth, and produces better pixel-level attention to the feature map, so that it has a good detection effect when targeting small targets on the surface of microelectronic devices. Secondly, during the model training process, the Inner-CIoU loss is introduced when calculating the loss, which is conducive to better locating the defects of small targets. In summary, the present invention can effectively solve the problem of surface defect detection of microelectronic devices and greatly improve the detection efficiency and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of steps of an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of the network structure of a microelectronic device surface defect detection model in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of the network structure of a CBS module in a microelectronic device surface defect detection model according to an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of the network structure of the RepNCSPELAN4 module in the microelectronic device surface defect detection model in an embodiment of the present invention;

[0019] Figure 5 Schematic diagram of the network structure of the ELAN-SPP module in the microelectronic device surface defect detection model in an embodiment of the present invention;

[0020] Figure 6 Schematic diagram of the network structure of the EMA module in the microelectronic device surface defect detection model in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The embodiment of the present invention provides a microelectronic device surface defect detection method based on improved YOLOv9, referring to Figure 1 As shown, the method includes:

[0023] Step 1: Collect surface images of microelectronic devices with and without surface defects, classify and annotate the defects respectively, and form a sample set of surface images of microelectronic devices;

[0024] Step 2: Perform data enhancement on the microelectronic device surface image sample set to obtain an expanded data set, and divide the expanded data set into a training set, a validation set, and a test set;

[0025] Step 3: Introduce an efficient multi-scale attention mechanism module into the backbone network of the YOLOv9 model to build a microelectronic device surface defect detection model based on the improved YOLOv9;

[0026] Step 4: Use the training samples in the training set to train the microelectronic device surface defect detection model based on the improved YOLOv9 to obtain an optimized microelectronic device surface defect detection model;

[0027] Step 5: Obtain the image of the microelectronic device to be inspected, input it into the optimized microelectronic device surface defect detection model for detection, and output the detection classification result.

[0028] In some embodiments, in step 1, after defect classification and annotation of the surface image of the microelectronic device, the surface image of the microelectronic device is divided into five defect categories including scratches, abrasions, stains, cracks and defects, and the microelectronic device images containing the five defect categories are used as positive samples, and the surface images of the microelectronic device without surface defects are used as negative samples. The positive sample set and the negative sample set together constitute a microelectronic device surface image sample set.

[0029] For example, a high-speed industrial camera is used to collect surface images of different types of microelectronic devices, totaling 500 images constituting an image dataset D, of which 250 images with surface defects are used as positive samples, and the images are labeled accordingly according to different defect types, including five categories: scratches, abrasions, stains, cracks, and incompleteness, with 50 images in each category and an image size of 640*640. 250 images without surface defects are used as negative samples.

[0030] In some embodiments, in step 2, data enhancement of the microelectronic device surface image sample set specifically includes random cropping, random scaling, and random arrangement of the microelectronic device images.

[0031] For example, the image dataset D is expanded by random cropping, random scaling, and random arrangement to generate a more diverse and representative surface defect image dataset Da. The expanded training dataset Da has 5,000 images, and Da is divided into a training set Dt, a validation set Dv, and a test set Dte in a ratio of 7:2:1.

[0032] In some embodiments, in step 3, the construction process of the microelectronic device surface defect detection model based on the improved YOLOv9 is specifically as follows:

[0033] The second output end of the RepNCSPELAN4 module of the last layer in the YOLOv9 backbone network is connected to the input end of the EMA module, and the output end of the EMA module is connected to the input end of the ELAN-SPP module in the network Head.

[0034] Figure 2 Schematic diagram of the network structure of the microelectronic device surface defect detection model in an embodiment of the present invention.

[0035] Figure 2 In the embodiment of the present invention, the EMA module is introduced into YOLOv9, that is, the EMA module is connected in series between the fourth RepNCSPELAN4 module of the backbone and the ELAN-SPP module of the head. The microelectronic device surface defect detection model based on the improved YOLOv9 includes: a backbone network backbone, a head, a multi-level auxiliary information structure and an auxiliary reversible branch, wherein the auxiliary reversible branch and the multi-level auxiliary information are only turned on in the training phase and turned off in the reasoning process.

[0036] The Backbone is the main part of the network and is responsible for extracting the features of the input image. The network structure of the Backbone includes: a Silence module, a first CBS module, a second CBS module, a first RepNCSPELAN4 module, a third CBS module, a second RepNCSPELAN4 module, a fourth CBS module, a third RepNCSPELAN4 module, a fifth CBS module, a fourth RepNCSPELAN4 module and an EMA module.

[0037] The Head is responsible for generating the final detection result. The Head adopts a decoupled head structure to separate the classification and detection modules. The network structure of the Head includes: an ELAN-SPP module, the input end of the ELAN-SPP module is connected to the output end of the EMA module, the output end of the ELAN-SPP module is connected to the first branch, the first branch includes the first upsampling, the first combination), the first output of the fifth RepNCSPELAN4 module, the second upsampling second combination), the sixth RepNCSPELAN4 module, the sixth CBS module, the third combination), the seventh RepNCSPELAN4 module, the seventh CBS module, the fourth combination), the eighth RepNCSPELAN4 module; the output end of the ELAN-SPP module is connected to the second branch, the second branch is connected to the fourth combination); the second output of the fifth RepNCSPELAN4 module and the third combination.

[0038] The specific structure of the auxiliary reversible branch structure includes in sequence: a Silence module, an eighth CBS module, a ninth CBS module, a ninth RepNCSPELAN4 module, a tenth CBS module, a first CBFuse module, a tenth RepNCSPELAN4 module, an eleventh CBS module, a second CBFuse module, an eleventh RepNCSPELAN4 module, a twelfth CBS module, a third CBFuse module, and a twelfth RepNCSPELAN4 module.

[0039] The specific structure of the multi-level auxiliary information structure includes: a first CBLinear module, a second CBLinear module and a third CBLinear module. The output end of the second RepNCSPELAN4 module of the Backbone is connected to the first CBLinear module; the output end of the third RepNCSPELAN4 module of the Backbone is connected to the second CBLinear module; the output end of the fourth RepNCSPELAN4 module of the Backbone is connected to the third CBLinear module; the outputs of the first, second and third CBLinear modules are all input into the first CBFuse module of the auxiliary reversible branch; the outputs of the second and third CBLinear modules are all input into the second CBFuse module of the auxiliary reversible branch; the output of the third CBLinear module is separately input into the third CBFuse module of the auxiliary reversible branch.

[0040] The Silence module does not perform any operation on the input image. Its function is to return the input image variable to facilitate the network's call to the input variable.

[0041] Figure 3 Schematic diagram of the network structure of the CBS module in the microelectronic device surface defect detection model in an embodiment of the present invention. Figure 3 In the figure, the specific structure of the CBS module includes: convolution layer Conv, batch normalization layer BN and activation function layer, where the activation function is SiLU.

[0042] Figure 4 Schematic diagram of the network structure of the RepNCSPELAN4 module in the microelectronic device surface defect detection model in an embodiment of the present invention. Figure 4In the example, the RepNCSPELAN4 module includes a three-layer structure: the RepNCSPELAN4 module, the RepNCSP module, and the RepNBottleNeck module. The RepNCSPELAN4 module is connected to the first CBS module and the Chunk(2,1) module in sequence, and the Chunk(2,1) module indicates that the input tensor is divided into two blocks along the vertical direction. The first output branch of the Chunk(2,1) module is connected to the first RepNCSP module, the second CBS module, the second RepNCSP module, and the third CBS module in sequence, and the third CBS module is connected to the combined path; the output of the second CBS module is connected to the combined path; the second output of the Chunk(2,1) module is connected to the combined path; the third output of the Chunk(2,1) module is connected to the combined path; the combined path is spliced ​​to the fourth CBS module through Concat. The RepNCSP module includes two output branches, wherein the first output branch is connected to the first CBS module and N RepNBottleNeck modules in sequence, and connected to the first junction; wherein the second output branch is connected to the second CBS module, and the output end of the second CBS module is connected to the first junction; the first junction is connected to the third CBS module through concat splicing, and the output end of the third CBS module is connected to the RepConvN module. The RepConvN module includes two output branches, wherein the first output branch is connected to the first CBS module, and its output end is connected to the junction; the other output branch is connected to the second CBS module, and its output end is connected to the junction, and the junction is superimposed by add. The RepNBottleNeck module includes two output branches, wherein the first output branch is connected to the RepConvN module and the CBS module in sequence, and the outputs of the CBS modules are connected to the junction; wherein the first output branch is connected to the junction; the junction is superimposed by add.

[0043] Figure 5 Schematic diagram of the network structure of the ELAN-SPP module in the microelectronic device surface defect detection model in an embodiment of the present invention. Figure 5 In the ELAN-SPP module, the first output branch of the first CBS module is connected to the first Maxpool module, the second Maxpool module and the third Maxpool module in sequence, and the output end of the third Maxpool module is connected to the combined road; the second output branch of the first CBS module is connected to the combined road; another output branch of the first Maxpool module is connected to the combined road; another output branch of the second Maxpool module is connected to the combined road; the combined road is spliced ​​by Concat and connected to the second CBS module.

[0044] Figure 6 Schematic diagram of the network structure of the EMA module in the microelectronic device surface defect detection model in an embodiment of the present invention. Figure 6In the embodiment, the EMA module includes a Groups module, a first branch of the Groups module is connected to an X Avg Pool module, and an output end thereof is connected to a combiner; a second branch of the Groups module is connected to a Y Avg Pool module, and an output end thereof is connected to a combiner; the combiner is spliced ​​by Concat and connected to a Conv module, the Conv module is respectively connected to two layers of activation modules (which include a Sigmoid function), and output ends thereof are respectively connected to a Re-weight module, a GroupNorm module, an Avg Pool module, a first Softmax module, and a first Matmul module in sequence, and an output end of the Matmul module is connected to the combiner; a third branch of the Groups module is connected to a Conv module, a first output branch of the Conv module is connected to a Matmul module in a second branch of the Groups module, a second output branch of the Conv module is respectively connected to an Avg Pool module, a second Softmax module, and a second Matmul module in sequence, and an output end thereof is connected to the combiner; the combiner is connected by Add, and the combiner is connected to a Sigmoid module, and an output end thereof is connected to a Re-weight module; a fourth branch of the Groups module is connected to a Re-weight module.

[0045] The input microelectronic device surface image is grouped by the feature map feature map1 output by the fourth RepNCSPELAN4 module of the YOLOv9 backbone. The size of map1 is C*H*W, which is grouped into g sub-feature maps, and the size of each sub-feature map is (C / / g)*H*W. Each sub-feature is processed through different branches, and two branches perform global pooling operations in the H dimension and W dimension respectively to obtain feature maps of (C / / g)*1*W and (C / / g)*H*1, which are concatenated and re-convolved to obtain a feature map of (C / / g)*1*(H+W). Then, the output features of this branch are modulated by the sigmoid function and normalization operation to capture and strengthen key features while suppressing unimportant information; one branch performs feature extraction through 3*3 convolution, and the size of the feature map remains unchanged; then, the modulated features are merged through a cross-dimensional interaction module, which can capture pixel-level pairwise relationships, thereby further enhancing the model's understanding and processing capabilities of features; finally, after the final sigmoid adjustment, the output feature map is used to enhance or weaken the original input features, so that the final output feature map has the same size as the original input, that is, C*H*W.

[0046] In some embodiments, in step 4, the microelectronic device surface defect detection model based on the improved YOLOv9 is trained using the training samples in the training set, and the specific training process is:

[0047] S401: Obtain training samples from the training set, input the training samples into the backbone network of YOLOv9, perform feature extraction, and obtain image features;

[0048] S402: Input the extracted image features into the head network part of YOLOv9 for classification judgment. The head network part includes a classification head and a regression head. The classification head is responsible for judging the target category to which each anchor point in the image belongs and obtaining a classification result. The regression head is responsible for adjusting the position and size of the anchor point and obtaining a bounding box regression result to more accurately locate the target.

[0049] S403: Based on the classification results and bounding box regression results, calculate the total loss between them and the true label of the input training sample point The total loss Including classification loss and regression loss; the total loss obtained by calculation Perform back propagation and adjust the model parameters according to the loss gradient. Repeat the above steps for iterative training. In each iteration, use new training samples for forward propagation, loss calculation and back propagation until the loss function converges, obtain and save the final model parameters, and obtain the optimized microelectronic device surface defect detection model.

[0050] During the model training phase, the auxiliary reversible branch structure is used to generate reliable gradients for updating network parameters. However, since the auxiliary reversible branch structure brings a lot of computational costs, it is only used in the training mode network structure and is turned off during the inference process.

[0051] In order to avoid the deep feature pyramid losing the different depth information of the predicted target object, the complete information of each main branch is retained through a multi-level auxiliary information structure to learn target prediction of various sizes. This structure is mainly to supplement the feature information of the auxiliary reversible branch structure, so it is only used in the training mode network structure.

[0052] During the training process, the model automatically learns how to extract useful features from the original image and classify defects based on these features. To speed up the training process, an iterative training method is used. After the final training, a deep learning detection model with optimal parameters (i.e., an optimized microelectronic device surface defect detection model) is obtained.

[0053] The Inner-CIoU Loss loss function is introduced in the model training process. The scale factor is used to control the generation of auxiliary borders of different scales for calculating the loss. The model parameters are optimized through backpropagation of the loss to improve the detection accuracy.

[0054] In some embodiments, in step 4, during the training process of the microelectronic device surface defect detection model based on the improved YOLOv9, the loss function used includes two branches, namely, classification loss and regression loss, as shown below:

[0055]

[0056] in, represents the total loss of the microelectronic device surface defect detection model, represents the boundary regression loss between the model output detection classification result image and its corresponding input image, w1 is the first weight, and w1 represents the weight of the boundary regression loss in the total loss; represents the classification loss between the model output detection classification result image and its corresponding input image, w2 is the second weight, and w2 represents the weight of the classification loss in the total loss.

[0057] Use Binary CrossEntropy (BCE) loss to calculate category classification loss The specific expression is:

[0058]

[0059] BCE Loss(p,y) =-(y×log(p)+(1-y)×log(1-q)) (3)

[0060] Among them, y represents the true label value of the input image, and p represents the predicted label value of the detection and classification result image.

[0061] The border regression loss is calculated using the Distribution Focal Loss (DFL) and the Inner-Complete Intersection over Union Loss (IOU, Inner-CIoU Loss) The specific expression is:

[0062]

[0063] DFL(δ i ,δ i+1 )=-((y i+1 -y)log(δ i )+(yy i )log(δ i+1 )) (5)

[0064] Among them, y represents the true label value of the input image, y i ,y i+1Represents the probability of optimizing the left and right positions near y, where y i represents the i-th label; y i ,y i+1 It can be understood that DFL optimizes the probabilities of the two positions closest to the label, one on the left and one on the right, in a form similar to cross entropy, so that the network can quickly focus on the distribution of the neighboring area of ​​the target position.

[0065] The definition of Inner-IoU is as follows:

[0066]

[0067]

[0068] uion=(w gt *h gt )*(ratio) 2 +(w*h)*(ratio) 2 -inter (15)

[0069]

[0070] Among them, ratio represents the scale factor, represents the center point of the ground-truth box, Represents the horizontal coordinate of the center point of the real frame, Represents the ordinate of the center point of the real frame, Indicates the value after the horizontal coordinate of the center point of the real frame is reduced by a certain ratio. It indicates the value after the horizontal coordinate of the center point of the real frame is enlarged by a certain ratio. It means the value after the vertical coordinate of the center point of the real frame is reduced by a certain ratio. Indicates the value of the ordinate of the center point of the real frame after it is enlarged by a certain ratio, b l Indicates the value of the horizontal coordinate of the center point of the anchor box after it is reduced by a certain ratio, b r Indicates the value of the horizontal coordinate of the center point of the anchor box after it is enlarged by a certain ratio, b t Indicates the value of the vertical coordinate of the center point of the anchor box after it is reduced by a certain ratio, b b Indicates the value of the vertical coordinate of the center point of the anchor box after it is enlarged by a certain ratio, w gt Indicates the width of the real box, h gt represents the height of the real box, w represents the width of the predicted box, h represents the height of the predicted box, and x c Indicates the horizontal coordinate of the center point of the anchor box, y c represents the ordinate of the center point of the anchor box, inter represents the intersection area between the predicted bounding box and the true bounding box, and union represents the union area between the predicted bounding box and the true bounding box. interrepresents the internal intersection-union ratio.

[0071] CIoU Loss considers three important parameters of the anchor box: overlapping area, center point distance, and aspect ratio. In this way, the predicted box will be more consistent with the real box. Inner-CIoU Loss inherits some characteristics of CIoU Loss and has its own characteristics.

[0072]

[0073] in, represents the complete intersection-over-union loss, CIoU represents the complete intersection-over-union (CIoU), and IoU represents the conventional intersection-over-union (IOU); ρ 2 (b,b gt ) represents the square of the Euclidean distance between the two center points of the predicted and actual anchor boxes, c represents the diagonal length of the minimum circumscribed rectangle of the two rectangles, α and ν are the relationship between the aspect ratio,

[0074]

[0075] In the formula, ν represents the relative proportional consistency value between the predicted bounding box and the true bounding box, α represents the weight coefficient, and w gt Indicates the width of the real box, h gt represents the height of the real box, w represents the width of the predicted box, and h represents the height of the predicted box.

[0076] After completing the training of the microelectronic device surface defect detection model based on the improved YOLOv9, the optimized microelectronic device surface defect detection model is obtained. The optimized microelectronic device surface defect detection model is tested and verified using the validation set and the test set. During the model testing and verification stage, the auxiliary reversible branch structure and the multi-level auxiliary information structure are turned off. According to the results of the test and verification, the number of training samples and the number of times the detection model is trained are adjusted to obtain the microelectronic device surface defect detection model with the optimal parameters.

[0077] In some embodiments, in step 5, after the image of the microelectronic device to be inspected is inspected by the optimized microelectronic device surface defect detection model, the output detection classification result image includes the defect type of each microelectronic device surface defect, and the position of each defect in the image is marked.

[0078] The optimized microelectronic device surface defect detection model is applied to the defect detection of microelectronic device surface images. In the actual detection stage, the auxiliary reversible branch structure and the multi-level auxiliary information structure are closed to obtain the actual surface image of the microelectronic device to be detected, which is input into the optimized microelectronic device surface defect detection model, and the features in the image are automatically extracted through the backbone network backbone; the extracted image features are input into the classifier, i.e., the head network, for classification and judgment of whether there are defects; then the feature map of the convolution layer is visualized by mirroring to achieve accurate positioning of the defects; the detection results are output in the form of images for users to view and analyze, and the output results include the type and location information of the defects.

[0079] Experimental verification:

[0080] The network model parameter configuration of the present invention is summarized in Table 1:

[0081] Table 1 Network model parameter configuration table

[0082] Configuration parameter Optimizer SGD Base learning rate 0.01 Base weight decay 0.0005 Optimizer momentum 0.937 Batch size 128 Learning rate schedule Linear Training epochs 500 Warmup iterations Max(1000,3*iters_per_epochs) Input size 640*640 EMA decay 0.9999

[0083] The performance of the improved YOLOv9 model is objectively evaluated. The commonly used precision, recall, and mean average precision (mAP) are used as evaluation criteria. The model is subjected to an ablation experiment. The surface defect detection performance results are shown in Table 2.

[0084] Table 2 Ablation experiment based on improved YOLOv9 algorithm

[0085]

[0086]

[0087] It can be seen from Table 2 that the microelectronic device surface defect detection method based on improved YOLOv9 proposed in the present invention (when Inner-CIoU ratio = 0.70) improves the precision, recall, mAP0.5 average precision, and mAP0.5:0.95 average precision by 0.7%, 1.7%, 1.2% and 3.1%, respectively, which has a good reference significance for the application of microelectronic device surface defect detection.

[0088] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0089] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting surface defects of microelectronic devices based on improved YOLOv9, characterized in that: include: Step 1: Collect surface images of microelectronic devices with and without surface defects, classify and annotate the defects respectively, and form a sample set of surface images of microelectronic devices; Step 2: Perform data enhancement on the microelectronic device surface image sample set to obtain an expanded data set, and divide the expanded data set into a training set, a validation set, and a test set; Step 3: Introduce an efficient multi-scale attention mechanism module into the backbone network of the YOLOv9 model to build a microelectronic device surface defect detection model based on the improved YOLOv9; Step 4: Use the training samples in the training set to train the microelectronic device surface defect detection model based on the improved YOLOv9 to obtain an optimized microelectronic device surface defect detection model; Step 5: Obtain the image of the microelectronic device to be inspected, input it into the optimized microelectronic device surface defect detection model for detection, and output the detection classification result.

2. According to a method for detecting surface defects of microelectronic devices based on improved YOLOv9 according to claim 1, it is characterized in that: In step 1, after defect classification and annotation of the surface images of microelectronic devices, the surface images of microelectronic devices are divided into five defect categories including scratches, abrasions, stains, cracks and defects. The microelectronic device images containing the five defect categories are used as positive samples, and the surface images of microelectronic devices without surface defects are used as negative samples. The positive sample set and the negative sample set together constitute a microelectronic device surface image sample set.

3. According to a method for detecting surface defects of microelectronic devices based on improved YOLOv9 according to claim 1, it is characterized in that: In step 2, data enhancement is performed on the microelectronic device surface image sample set, specifically including random cropping, random scaling, and random arrangement of the microelectronic device images.

4. According to a method for detecting surface defects of microelectronic devices based on improved YOLOv9 according to claim 1, it is characterized in that: In step 3, the construction process of the microelectronic device surface defect detection model based on improved YOLOv9 is specifically as follows: The second output end of the RepNCSPELAN4 module of the last layer in the YOLOv9 backbone network is connected to the input end of the EMA module, and the output end of the EMA module is connected to the input end of the ELAN-SPP module in the network Head.

5. A microelectronic device surface defect detection method based on improved YOLOv9 according to claim 1, characterized in that: In step 4, the surface defect detection model of microelectronic devices based on improved YOLOv9 is trained using the training samples in the training set. The specific training process is as follows: S401: Obtain training samples from the training set, input the training samples into the backbone network of YOLOv9, perform feature extraction, and obtain image features; S402: Input the extracted image features into the head network part of YOLOv9 for classification judgment. The head network part includes a classification head and a regression head. The classification head judges the target category to which each anchor point in the image belongs according to the input image features to obtain a classification result. The regression head adjusts the position and size of each anchor point to obtain a bounding box regression result. S403: Based on the classification results and bounding box regression results, calculate the total loss between them and the true label of the input training sample The total loss Including classification loss and regression loss, the total loss is calculated Back propagation is performed, model parameters are adjusted according to the loss gradient, and training is iterated until the loss function converges to obtain an optimized microelectronic device surface defect detection model.

6. A microelectronic device surface defect detection method based on improved YOLOv9 according to claim 1 or 5, characterized in that: In step 4, during the training of the microelectronic device surface defect detection model based on the improved YOLOv9, the loss function used is specifically: DFL(d i ,d i+1 )=-((y i+1 -y)log(δ i )+(yy i )log(δ i+1 )), in, represents the total loss of the microelectronic device surface defect detection model, represents the boundary regression loss between the model output detection classification result image and its corresponding input image, w1 is the first weight, and w1 represents the weight of the boundary regression loss in the total loss; represents the classification loss between the model output detection classification result image and its corresponding input image, w2 is the second weight, and w2 represents the weight of the classification loss in the total loss, DFL(δ i ,δ i+1 ) represents the distribution focal loss, Among them, y represents the true label value of the input image, y i ,y i+1 Respectively represent the probability of optimizing the left and right positions near y, y i represents the i-th label; represents the internal complete intersection-combination loss, represents the complete intersection-over-union loss, IoU represents the intersection-over-union ratio, IoU inter represents the internal intersection-union ratio.

7. The method for detecting surface defects of microelectronic devices based on improved YOLOv9 according to claim 1, characterized in that: In step 5, after the image of the microelectronic device to be inspected is inspected by the optimized microelectronic device surface defect detection model, the output detection classification result image includes the defect type of each microelectronic device surface defect, and the position of each defect in the image is marked.

Images