Micro-nano structure defect identification method, apparatus and device, and storage medium

Abstract

The invention relates to the technical field of defect identification, in particular to a micro-nano structure defect identification method and device, equipment and a storage medium. According to the method, a real defect sample picture is adopted to guide deep convolutional generative adversarial network learning to convert a noise sample vector into a defect picture so as to obtain a defect picture generator, and then the defect picture generator is used for generating a large number of virtual defect pictures; the virtual defect pictures are fused into the original images in the original image set of the micro-nano structure, so that the number of the defect pictures contained in the original image set is increased, more defect images are obtained, the model is trained based on a large number of defect images, and the precision of the defect recognition model obtained through training can be improved. A convolution strategy for optimizing a backbone network is provided based on a YOLOv10 model, a neck network is improved, the training efficiency and the detection speed of the model are remarkably improved by decoupling extraction of core features and redundant features, and meanwhile efficient utilization of computing resources is achieved.

Description

Technical Field

[0001] The present invention relates to the field of defect recognition technology, and in particular to a micro-nano structure defect recognition method, device, equipment and storage medium. Background Art

[0002] Micro-nano structures have been widely used in precision machining, preparation, multifunctionality and the expansion of new application fields due to their superior physical and chemical properties. They are widely used in optics (such as microlenses, integrated optics and optical metamaterials), mechanics (such as micro-electromechanical system devices), electronics (such as integrated circuits) and biomedicine (such as artificial biological organs) and other fields. If defects occur in micro-nano structures, the performance of the micro-nano structures will be affected, so it is necessary to detect and identify the defects of the micro-nano structures. The defect recognition model of convolutional neural network is used to detect the defects of micro-nano structures, and the convolutional neural network needs to be trained with a large number of samples before it can identify defects. However, the existing samples of micro-nano structures with defects are scarce, resulting in low accuracy of the trained model.

[0003] In summary, the existing technology reduces the model accuracy due to insufficient training samples.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a micro-nanostructure defect recognition method, device, equipment and storage medium, which solves the problem that the prior art reduces the model accuracy due to insufficient training samples.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for identifying micro-nanostructure defects, comprising:

[0008] The original image in the original image set of the micro-nano structure is cropped to obtain a defect sample image; and a noise sample vector is obtained, and based on the defect sample image and the noise sample vector, a deep convolutional generative adversarial network is trained to obtain a defect image generator;

[0009] Acquire a noise vector, input the noise vector into the defect image generator to obtain a virtual defect image, and fuse the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures;

[0010] Based on the defect enhanced image set, a defect recognition model is trained to obtain a defect recognition trained model;

[0011] A measured image with a micro-nano structure is acquired, and the defect recognition trained model is applied to the measured image to obtain a defect recognition result.

[0012] In one implementation, the deep convolutional generative adversarial network includes a generator and a discriminator. Based on the defect sample image and the noise sample vector, the deep convolutional generative adversarial network is trained to obtain a defect image generator, including:

[0013] Input the noise sample vector into the generator to obtain a virtual sample image output by the generator;

[0014] Inputting the virtual sample image and the defect sample image into the discriminator to obtain a loss function output by the discriminator, wherein the loss function is used to characterize difference information between the virtual sample image and the defect sample image;

[0015] Based on the loss function, the generator and the discriminator are trained alternately, and the generator that has completed the training is used as a defect image generator.

[0016] In one implementation, the number of network layers of the generator and the number of network layers of the discriminator are both related to the resolution of the defect image that the generator needs to generate.

[0017] In one implementation, the virtual defect image is merged into the original image to obtain a defect enhanced image set of the micro-nano structure, including:

[0018] Determine a hole defect virtual image, a stripe aggregation defect virtual image, and a nanocolumn defect virtual image in the virtual defect image;

[0019] An image fusion method based on a sliding window is applied to the virtual image of the hole defect and the original image, an image fusion method based on a sliding window is applied to the virtual image of the stripe aggregation defect and the original image, and an image fusion method based on a sliding window is applied to the virtual image of the nanocolumn defect and the original image, so as to obtain a defect enhanced image set of the micro-nano structure.

[0020] In one implementation, the defect recognition model is an improved YOLOv10 model, and the convolution layer in the spatial pyramid pooling module of the improved YOLOv10 model is a depth-separable convolution layer. In one implementation, the feature extraction structure in the neck of the YOLOv10 model includes a front convolution layer, a segmentation layer, a bottleneck structure group, a splicing layer, and a rear convolution layer connected in sequence, and the bottleneck structure group includes several improved bottleneck structures, and each improved bottleneck structure includes a first Ghost convolution layer, a batch normalization layer, a ReLU activation function layer, a second Ghost convolution layer, and a residual connection layer connected in sequence.

[0021] In one implementation, applying the defect recognition trained model to the measured image to obtain a defect recognition result includes:

[0022] The defect recognition trained model is applied to the measured image to obtain the defect type, defect size and defect position, and the defect type, defect size and defect position are used as the defect recognition result.

[0023] In a second aspect, an embodiment of the present invention further provides a micro-nano structure defect identification device, wherein the device includes the following components:

[0024] The first training module is used to crop the original image in the original image set of the micro-nano structure to obtain a defect sample image; and obtain a noise sample vector, and based on the defect sample image and the noise sample vector, train a deep convolutional generative adversarial network to obtain a defect image generator;

[0025] An image expansion module, used for obtaining a noise vector, inputting the noise vector into the defect image generator to obtain a virtual defect image, and fusing the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures;

[0026] A second training module is used to train a defect recognition model based on the defect enhanced image set to obtain a defect recognition trained model;

[0027] The recognition module is used to obtain a measured image with a micro-nano structure, and apply the defect recognition trained model to the measured image to obtain a defect recognition result.

[0028] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and a micro-nanostructure defect identification program stored in the memory and executable on the processor, and when the processor executes the micro-nanostructure defect identification program, the steps of the above-mentioned micro-nanostructure defect identification method are implemented.

[0029] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a micro-nanostructure defect recognition program is stored. When the micro-nanostructure defect recognition program is executed by a processor, the steps of the micro-nanostructure defect recognition method described above are implemented.

[0030] Beneficial effect: The present invention uses real defect sample images to guide deep convolutional generative adversarial network learning to convert noise sample vectors into defect images to obtain a defect image generator, and then uses the defect image generator to generate a large number of virtual defect images, which are integrated into the original images in the original image set of the micro-nano structure to expand the number of defect images contained in the original image set to obtain more and more balanced defect images. Based on a large number of high-quality defect image training models, the accuracy of the defect recognition model obtained by training can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the overall flow chart of the present invention;

[0032] Figure 2 Schematic diagram of three defects in an embodiment of the present invention;

[0033] Figure 3 This is a structural diagram of DCGAN based on sliding window image clipping, fusion and improvement in an embodiment of the present invention;

[0034] Figure 4 is a structural diagram of a generator and a discriminator in an embodiment of the present invention;

[0035] Figure 5 This is a structural diagram of a target detection model based on the improvement of YOLOv10 in an embodiment of the present invention;

[0036] Figure 6 4 is a structural diagram of a DwSPPF deep separable spatial pyramid pooling module in an embodiment of the present invention;

[0037] Figure 7 Schematic diagram of the operation of depthwise separable convolution in an embodiment of the present invention;

[0038] Figure 8 Schematic diagram of a C2fGhost feature extraction module in an embodiment of the present invention;

[0039] Fig. 9 Schematic diagram of a Ghost convolution module in an embodiment of the present invention;

[0040] Fig.10 It is a schematic diagram comparing the convergence speeds of various models after improving the spatial pyramid pooling structure using different optimization methods in the embodiment;

[0041] Fig.11 It is a schematic diagram comparing the optimized average accuracy and parameter amount of each model after improving the spatial pyramid pooling structure by different optimization methods in the embodiment;

[0042] Fig.12Improved combined parameters and accuracy scatter plots for each module of the neck network in the embodiment;

[0043] Fig.13 A structural diagram of a micro-nanostructure defect identification device provided by the present invention;

[0044] Fig.14 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Research has found that micro-nano structures have been widely used in precision processing, preparation, multifunctionality and the expansion of new application fields due to their superior physical and chemical properties. They are widely used in many fields such as optics (such as microlenses, integrated optics and optical metamaterials), mechanics (such as micro-electromechanical system devices), electronics (such as integrated circuits) and biomedicine (such as artificial biological organs). If defects occur in micro-nano structures, the performance of the micro-nano structures will be affected, so it is necessary to detect and identify the defects of the micro-nano structures. The defect recognition model of convolutional neural network is used to detect the defects of micro-nano structures, and the convolutional neural network needs to be trained with a large number of samples before it can identify defects.

[0047] In order to solve the above technical problems, the present invention provides a method, device, equipment and storage medium for identifying defects in micro-nano structures, which solves the problem that the existing technology reduces the accuracy of the model due to insufficient training samples and unbalanced samples. In specific implementation, the original image in the original image set of the micro-nano structure is first cropped to obtain a defect sample image; and a noise sample vector is obtained, and a deep convolutional generative adversarial network is trained based on the defect sample image and the noise sample vector to obtain a defect image generator; then a noise vector is obtained, and the noise vector is input into the defect image generator to obtain a virtual defect image, and the virtual defect image is fused into the original image to obtain a defect enhancement image set of the micro-nano structure; then, based on the defect enhancement image set, a defect recognition model is trained to obtain a defect recognition trained model; finally, a measured image with a micro-nano structure is obtained, and the defect recognition trained model is applied to the measured image to obtain a defect recognition result.

[0048] For example, the surface of a micro-injection molded part contains micro-nano structures, and images of the surface of the micro-injection molded part with defects are collected to obtain an original image set, which contains several original images. The area where the defects are located on each original image is cropped to obtain a defect sample picture, and the defect sample picture and the generated noise sample vector are input into a deep convolutional generative adversarial network. Under the guidance of the defect sample picture, the deep convolutional generative adversarial network learns how to convert the noise sample vector into a defect picture. When the deep convolutional generative adversarial network has the function of generating virtual pictures that are highly similar to real defect images, the deep convolutional generative adversarial network becomes a defect picture generator.

[0049] When a large number of defect images are needed to train the defect recognition model, since the defect image generator has learned how to generate defect images, the noise vector is input into the defect image generator, and the defect image generator can generate a large number of virtual defect images, extract virtual defect images and original images, randomly select the pasting position, and randomly rotate and scale the virtual defect images to increase the diversity of samples. Finally, the processed virtual defect image is pasted into the original image to achieve image fusion. This method expands and balances the image set to improve the accuracy of the model after training.

[0050] The micro-nanostructure defect recognition method of this embodiment can be applied to a terminal device, which can be a terminal product with an image processing function, such as a computer. Figure 1 As shown in , the micro-nanostructure defect identification method specifically includes the following steps:

[0051] S100, cropping the original image in the original image set of the micro-nano structure to obtain a defect sample image; and obtaining a noise sample vector, and training a deep convolutional generative adversarial network based on the defect sample image and the noise sample vector to obtain a defect image generator;

[0052] S200, obtaining a noise vector, inputting the noise vector into the defect image generator to obtain a virtual defect image, and fusing the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures;

[0053] S300, training a defect recognition model based on the defect enhanced image set to obtain a defect recognition trained model;

[0054] S400, obtaining a measured image with a micro-nano structure, and applying the defect recognition trained model to the measured image to obtain a defect recognition result.

[0055] In the first embodiment, the micro-nano structure is a surface micro-nano structure formed by micro-injection molding of polyurethane elastomer (TPU), and the specific process of forming the micro-nano structure is as follows:

[0056] Select a single-channel AAO template with a pore size range of [40nm, 450nm] and a pore depth range of [150nm, 20μm]; cut the AAO template according to the mold size and install it to the dynamic mold position. Secondly, put the TPU particles into the barrel and heat it to 230℃ to reach a molten state. Set the parameters of the Babyplast micro-precision injection molding machine to: mold temperature 30℃, injection speed 40mm / s and injection pressure 15MPa. With the holding time set to 5 seconds and the cooling time set to 8 seconds, the molten TPU is filled into the entire mold cavity and cooled and solidified. Subsequently, the mold is separated to remove the injection-molded TPU product, and the product is trimmed to ensure a smooth surface. Next, the AAO template is soaked in 10% NaOH solution for demolding, that is, the micro-injection molded part is removed from the mold cavity of the AAO template.

[0057] After the micro-nano structure is obtained through the above operation, a scanning electron microscope (SEM) is used to collect the surface image of the micro-nano structure. The surface images of several micro-nano structures constitute an original image set. Each original image in the original image set is a surface image of each micro-nano structure. The resolution of each original image is 1536×1103 pixels. The number of original images is 78, and all original images cover 619 defects. The defects on each original image are annotated using the open source tool Labelme, and a corresponding XML annotation file is generated. The defect types in this embodiment are as follows: Figure 2 As shown, Figure 2 Figure a is a hole defect. Figure 2 Figure b shows a strip-shaped aggregation defect. Figure 2 Figure c shows a nanocolumn defect. The original images with the above three defect types were made into training sets and validation sets in a ratio of 8:2.

[0058] This embodiment uses the following method to crop the original image:

[0059] When the size of the original image is 1536×1103 and the defect size of the original image is small, a sliding window algorithm with a window size of 200×200 and a step size of 100 is used to crop the defects on the original image to obtain a defect sample image; when the defect size of the original image is large, a sliding window algorithm with a window size of 400×400 and a step size of 200 is used to crop the defects on the original image to obtain a defect sample image. The defect sample image cropped in the above manner can retain the defect features as much as possible and avoid miscutting or missing important defect information during the cropping process.

[0060] This embodiment further enhances the cropped defect sample images by flipping, rotating, and mirroring to increase the number of training samples.

[0061] The structure of the deep convolutional generative adversarial network in this embodiment is as follows: Figure 3 As shown in Figure 2, the deep convolutional generative adversarial network includes a generator and a discriminator. The generator is Figure 3 The Generator in the discriminator is Figure 3 The specific structures of Generator and Discriminator are as follows Figure 4 As shown in the figure, the generator includes seven network layers, namely ConTranspose1, ConTranspose2, ConTranspose3, ConTranspose4, ConTranspose5, ConTranspose6 and ConTranspose7, which are connected in sequence; the discriminator includes seven network layers, namely Conv1, Conv2, Conv3, Conv4, Conv5, Conv6 and Conv7, which are connected in sequence. The seven network layers can avoid the defects of blurred boundaries, unrealistic textures, strong monotony and obvious noise points in the generated images. The parameters of each network layer are shown in Table 1:

[0062] Table 1

[0063]

[0064]

[0065] The noise sample vector in this embodiment is a randomly generated noise vector. In step S100, based on the defect sample image and the noise sample vector, a deep convolutional generative adversarial network is trained to obtain a defect image generator, which includes the following specific steps S101, S102, and S103:

[0066] S101, input the noise sample vector into the generator to obtain a virtual sample image output by the generator.

[0067] The noise sample vector is Figure 3 The Noise in the image is input into the generator, which converts the noise sample vector into a virtual sample image to simulate the defects on the micro-nano structure.

[0068] S102, inputting the virtual sample image and the defective sample image into the discriminator to obtain a loss function output by the discriminator, wherein the loss function is used to characterize difference information between the virtual sample image and the defective sample image.

[0069] The greater the difference between the virtual sample images generated by the generator and the real defect sample images, the greater the loss function; the smaller the difference between the virtual sample images generated by the generator and the real defect sample images, the smaller the loss function.

[0070] S103, based on the loss function, alternately train the generator and the discriminator, and use the trained generator as a defect image generator.

[0071] Alternately train the generator and discriminator, that is, fix the parameters of one and update the parameters of the other based on the loss function. The number of iterations iter and the total number of virtual sample images sum photos And the number of samples batched into the model each time during training size And the epoch of each training satisfies the following relationship:

[0072]

[0073] As the number of iterations increases, the quality of the images generated by the generator also improves, and the accuracy of the discriminator also improves. When the number of iterations reaches 4500, the virtual sample images generated by the generator are more similar to the real defect images, and the clarity of the virtual sample images is higher.

[0074] The real defect sample images in this embodiment include real hole defect images, real stripe aggregation defect virtual images, and real nanocolumn defect images, so as to realize the use of multiple types of defect images to train the generator, that is, using one defect type image to train the generator, the generator after training can have a large number of images of the defect, and using different types of defect images can train a generator that can generate different types of defect images. Therefore, the generator using this embodiment can solve the following technical problems:

[0075] There are significant differences in the number, scale and spatial position of various defects. The imbalance of sample categories will cause the network to have deviations in defect learning, which will reduce the model's ability to detect defects in a few categories, thereby affecting the detection accuracy and generalization ability of the model. Scale imbalance may cause the model to fail to effectively learn feature representations at different scales, resulting in the loss of some defect features, thus affecting the accuracy and speed of detection; while spatial imbalance may prevent the model from fully focusing on key areas or features, resulting in a decrease in defect detection capabilities in important areas, thus affecting detection accuracy.

[0076] Embodiment 2, based on embodiment 1, since embodiment 1 uses real hole defect images, real stripe aggregation defect virtual images, and real nanocolumn defect images to train the generator, the defect image generator obtained by training can generate virtual images for simulating hole defects, stripe aggregation defects, and nanocolumn defects. In this example, the step S200 of fusing the virtual defect image into the original image to obtain a defect enhanced image set of the micro-nano structure includes: applying a sliding window-based image fusion method to the hole defect virtual image and the original image, applying a sliding window-based image fusion method to the stripe aggregation defect virtual image and the original image, and applying a sliding window-based image fusion method to the nanocolumn defect virtual image and the original image to obtain a defect enhanced image set of the micro-nano structure.

[0077] The virtual image of hole defects is integrated into any original image in the original image set and at any position on any original image. The same processing is performed on the virtual image of strip aggregation defects and the virtual image of nanocolumn defects to balance the various types of defects in the original image set, so that the defect recognition model trained using the balanced original image set can recognize various defects.

[0078] This embodiment uses a sliding window-based image fusion method to paste the generated three types of virtual defect small images (virtual defect small images are virtual defect pictures, and the size of the image is 256×256) into large-sized micro-nano defect images (micro-nano defect images are original images in the original image set) to better match the size of the original data set and enhance the data set at the same time. There are a large number of small defect images in the original image set. In order to obtain more medium and large defect samples and solve the image scale imbalance and spatial imbalance, the following method is adopted:

[0079] Define the boundary of the sliding window, and define the window size of small-sized defects "Pothole" (Pothole is a hole defect) and "Fracture" (Fracture is a nanocolumn defect) as 300×300, the step size is 200, and the zoom range is controlled in (0.5,1.5); for large-sized defects "Stick" (Stick is a strip aggregation defect), the window size is 500×500, the step size is 300, and the zoom range is controlled in (1,2). Then, randomly select the position of the sliding window, and randomly rotate and scale the small image to increase the diversity of the sample. Finally, paste the processed small image into the large image (the large image is the original image) to achieve image fusion. This image fusion method based on the sliding window algorithm can effectively improve the diversity of the data and provide a more representative data set for subsequent model training.

[0080] Embodiment 3: This embodiment is based on Embodiment 1 or Embodiment 2. The defect recognition model in step S300 of this embodiment is an improved YOLOv10 model. The structure of the improved YOLOv10 model is as follows: Figure 5 As shown, Figure 5 Backbone is the backbone network of the improved YOLOv10 model, Neck is the neck network of the improved YOLOv10 model, and Head is the output part of the improved YOLOv10 model. The backbone network includes convolution (Conv), CSP Bottleneck with 2Convolutions (C2f), Selective Channel Down-sampling (SCDown), Deep Separable Spatial Pyramid Pooling (DwSPPF) and Pyramid Split Attention (PSA) modules. The spatial pyramid pooling (SPPF) module is used to fuse feature maps of different scales, thereby establishing associations between objects of different scales and effectively solving the problem of information description of regional associations. The structural diagram of the deep separable spatial pyramid pooling DwSPPF (DwSPPF is shown in the figure below) Figure 6 The convolution layer of DwConv in the figure is a depth-separable convolution layer DSC, which uses depth-separable convolution to reduce structural redundancy, reduce the number of model parameters and calculations, and maintain good accuracy. Figure 7 As shown in the figure, the structure of DSC consists of two parts: channel-by-channel convolution and point-by-point convolution. Channel-by-channel convolution applies a convolution kernel of a specific size to the feature map of each channel to perform a normal convolution operation to generate a new feature map. In this process, the number of feature maps remains unchanged, and each channel corresponds to only one convolution kernel. Point-by-point convolution uses a 1×1 convolution kernel to convolve the new feature map to adjust the output channel.

[0081] Assume that the DSC input data and output data The channel-by-channel convolution kernel is The point-by-point convolution kernel is c represents the number of input channels, n represents the number of output channels, k is the size of the convolution kernel, h and w are the height and width of the input data, and h' and w' are the height and width of the output data. In the standard convolution operation, the calculation formula for floating point operations (FLOPs) is:

[0082] F1=n×h′×w′×c×k×k

[0083] According to the channel-by-channel convolution operation, the calculation formula of FLOPs is:

[0084] F2=h′×w′×k×k×c

[0085] In point-by-point convolution, the convolution kernel K2 is used to generate a feature map with n channels, and its FLOPs is: F3 = h'×w'×c×n.

[0086] Therefore, the total FLOPs of depthwise separable convolution is:

[0087] F4=h'×w'×k×k×c+h'×w'×c×n

[0088] It can be obtained that the FLOPs ratio of depth-separable convolution to traditional standard convolution is:

[0089]

[0090] The final amount of calculation is reduced, and the reduction ratio is Depth-wise separable convolution reduces structural redundancy and reduces the number of module parameters and computation by decomposing the convolution operation into channel-by-channel convolution and point-by-point convolution. The convolution layer of the DwSPPF module uses a depth-wise separable convolution layer (DSC), which can improve the detection speed of the model while maintaining a high detection accuracy.

[0091] The C2fGhost module in the neck network of this embodiment is as follows Figure 8 As shown in Figure b, the C2fGhost module includes the front convolution layer, the segmentation layer (slipt), the bottleneck structure group, the splicing layer (concat), and the back convolution layer. Figure 8 In Figure b, the Conv on the left is the front convolution layer, and the Conv on the right is the back convolution layer. The bottleneck structure group includes the first bottleneck structure GhostBottleneck, the second bottleneck structure GhostBottleneck, and the nth bottleneck structure GhostBottleneck connected in series, wherein the input end of the first bottleneck structure GhostBottleneck is connected to the output end of Slipt, the output end of the nth bottleneck structure GhostBottleneck is connected to the input end of Concat, and the output end of Slipt is also connected to the input end of Concat.

[0092] The structure of each GhostBottleneck is as follows Figure 8As shown in Figure a, that is, GhostBottleneck includes a first GhostConv module, a second GhostConv module, and a residual connection layer connected in sequence. The first GhostConv module serves as an expansion layer to increase the number of channels of the input feature map; the second GhostConv module is used to reduce the number of channels to match the size of the shortcut. The input and output are connected between the two GhostConv modules through the shortcut. After the second GhostConv module, the ReLU activation function is no longer used, but batch normalization and ReLU non-linear activation are applied after other layers.

[0093] As Fig. 9 shown, let the input data of the ordinary convolution module be where c is the number of input channels, h and w are the height and width of the input data respectively. The operation of any convolution layer for generating a feature map with n channels can be expressed as:

[0094]

[0095] In the formula, * represents the convolution operation, b is the bias term, Y is the output feature map with n channels, f is the convolution filter in this layer, h' and w' are the height and width of the output feature map respectively, and k×k is the convolution kernel size of the convolution filter f. In this ordinary convolution operation, the calculation formula of FLOPs is:

[0096] F1 = n×h'×w'×c×k×k

[0097] As Figure 8 shown in Figure b, the Ghost module uses a small convolution filter f' to convolve the input core features to generate a preliminary feature map with m (m << n) channels:

[0098]

[0099] Then, a more inexpensive linear operation is performed on each channel of Y' to generate a feature map Y with n = m×s channels. The calculation formula of FLOPs for this process is:

[0100]

[0101] where s is the expansion factor in Ghost convolution, and d (d ≤ k) is the average kernel size in the linear operation process. Therefore, the theoretical acceleration ratio of calculating using GhostConv instead of ordinary convolution is:

[0102] The improvement in the theoretical speedup ratio shows that after the GhostConv module replaces the ordinary convolution, the calculation speed that can be obtained under the same calculation task is s times the original speed. Similarly, the compression ratio can be calculated as:

[0103]

[0104] It can be seen that the compression ratio is equal to the theoretical speedup ratio, both reaching s. With the Ghost module, the required number of parameters and the amount of calculation can be reduced to the original value. The above design significantly improves the computing efficiency.

[0105] Embodiment 4. This embodiment is based on embodiment 1 or embodiment 2 or embodiment 3. S400 in this embodiment includes the following specific steps: applying the defect recognition trained model to the measured image to obtain the defect type, defect size and defect location, and using the defect type, defect size and defect location as the defect recognition result.

[0106] That is, this embodiment can identify the defect type and defect size on the image and the position of the defect on the micro-nano structure in the image.

[0107] Embodiment 5, based on any one of Embodiments 1 to 4, uses average precision (AP) and mean average precision (mAP) as indicators. AP is used to comprehensively evaluate precision and recall, and calculates precision based on different recall rates and integrates precision.

[0108]

[0109] Among them, Precision(Recall) is the accuracy under the corresponding recall rate. Precision represents the proportion of samples predicted to be positive that are actually positive:

[0110]

[0111] Recall represents the proportion of samples that are actually positive that are predicted to be positive:

[0112]

[0113] Among them, TP (True Positive) is a true positive example, FP (False Positive) is a false positive example, and FN (False Negative) is a false negative example. mAP is used to measure the performance of the model in multi-category object detection tasks and is the average of APs of multiple categories.

[0114]

[0115] Where c is the number of categories, APc is the AP value of category c. mAP(@0.5) represents the average AP value of all categories detected by the model when Intersection over Union (IoU) is 0.5. IoU is used to measure the overlap between the predicted box and the true box, and its formula is:

[0116]

[0117] Among them, A is the area of ​​the predicted box and B is the area of ​​the real box.

[0118] In terms of evaluating the efficiency of model detection, the number of parameters, floating point operations (FLOPs) and Frames Per Second (FPS) are used as indicators. The number of model parameters is an important indicator for evaluating model complexity, which refers to the total number of all trainable parameters that need to be learned. The FPS of the model can be expressed as:

[0119]

[0120] Among them, FPS is the unit of frames per second, and IT represents the model inference time in milliseconds.

[0121] The following is a comparison experiment and ablation experiment designed on a deep learning workstation to demonstrate the effectiveness of the present invention in defect image expansion and the high performance of the defect recognition trained model. The configuration of the deep learning workstation is as follows:

[0122] The system is equipped with a 12th-generation Intel(R) Core(TM) i9-12900KS 3.40GHz processor, 64GB of memory, and an NVIDIA RTX A4000 graphics card with 16GB of video memory. The operating system is 64-bit Windows 11 Professional (version 10.0, number 22000), the programming environment is Python 3.8.18, and the machine learning framework uses PyTorch 2.2.1.

[0123] The YOLOv10n model is applied to defect enhancement image sets of various orders of magnitude. According to the defect recognition results output by the YOLOv10n model, the mAP (@0.5) indicator calculation of the YOLOv10n model is shown in Table 2:

[0124] Table 2

[0125]

[0126] As can be seen from Table 2, as the size of the dataset increases, the average accuracy of a single category and all categories increases. However, once the model has fully learned the features in the dataset, it cannot extract more information from the new data, which results in little difference in the accuracy of the training results between 1000 and 1200. If the dataset continues to increase from 1200, the overall accuracy may show a downward trend due to overfitting.

[0127] The influence of the defect image expansion method (i.e., BDE) of the present invention on the model accuracy is compared with the influence of no expansion, the influence of random crop defect image expansion method on the model accuracy, the influence of random horizontal flip defect image expansion method on the model accuracy, the influence of cut out defect image expansion method on the model accuracy, and the influence of random erasing defect image expansion method on the model accuracy, and the comparison results shown in Table 3 are obtained, wherein random crop means randomly cropping the image with a ratio of 0.6 to 1.0; random horizontal flip means randomly flipping the image horizontally with a probability of 0.5; cut out means randomly filling the image with a rectangular area with an area ratio of 0.3, and the filling content is zero pixels; random erasing means randomly filling the training set image with a rectangular area with an area ratio of 0.2 to 0.4, and the filling content is the average pixel.

[0128] Table 3

[0129]

[0130] As can be seen from Table 3, the average accuracy of the BDE method of the present invention trained on a dataset of 400 is improved by 18.5% compared with the original dataset. This significant improvement verifies the effectiveness of the data enhancement method.

[0131] The DwSPPF module of the present invention is compared with the SPPF Ghost module and the SPPF module, and the following is obtained: Fig.10 The comparison results of training efficiency shown in Fig.11 The performance comparison results are shown. Fig.10 and Fig.11It can be seen that compared with the original SPPF module, although the SPPF Ghost module has 77,000 fewer parameters, it converges more slowly during training and has a 0.6% decrease in the mAP(@0.5) indicator. This may be because the Ghost convolution in the SPPF Ghost module is not able to fully extract the features of the input image, resulting in insufficient feature representation of subsequent layers. In contrast, when using the DwSPPF module, the model achieved a 1% improvement in mAP(@0.5) and reduced the number of parameters by 163,000. This shows that through the decomposition operation of the deep separable convolution, the DwSPPF module significantly reduces the computational complexity and model parameters, enabling the model to efficiently complete deep learning tasks in a resource-constrained environment.

[0132] After the backbone network uses the DwSPPF module, the Conv module in the original neck network is replaced with GhostConv or DwConv, and the C2f module is replaced with C2fDw or C2fGhost. Fig.12 The combined experiment shown. The data sets used for training are all 1200 images after BDE data enhancement. Experiment 1 is an improved model trained by the original YOLOv10n. Table 4 shows the experimental results of different convolutional module combinations. The mAP of the models in Experiments 2, 4, 5 and 7 have increased after optimization. Taking Experiment 1 as the control group, Experiments 3, 6 and 9 used the C2fDw module. Although the number of parameters of the model can be significantly reduced, the feature extraction ability is relatively weak, which significantly reduces the mAP of the model. Although the mAP of the models in Experiments 4 and 7 has increased slightly, the reduction in the number of parameters is not obvious. In contrast, Experiment 5 obtained the best results. While the mAP (@0.5) increased by 1.1%, the number of parameters was significantly reduced by 184,000.

[0133] Table 4

[0134]

[0135]

[0136] In order to evaluate the contribution of each module to the performance of the overall model (the overall model is composed of a defect image generator and a defect recognition trained model), the YOLOv10 model was combined with each module to perform ablation experiments on the basis of image enhancement. As can be seen from Table 5, the addition of each module has improved the performance of YOLOv10. The DwSPPF module reduces the number of model parameters by 163,000, and the lightweight effect is the most obvious. In addition, the GhostConv module increases the mAP of the model by 1.4%, which has the most obvious improvement on the accuracy of the model. The combination of YOLOv10n+DwSPPF+GhostConv+C2fGhost is the YOLO-MD lightweight model designed in this paper. Compared with the original YOLOv10n model, the mAP (@0.5) of the YOLO-MD model increased by 2.1%, and the number of parameters decreased by 347,000. The above results verify the effectiveness and superiority of the improved model in TPU micro-nanostructure defect detection.

[0137] Table 5

[0138]

[0139] The YOLO-MD model is compared with the two-stage target detection model Faster R-CNN and five one-stage target detection models SSD, YOLOv3, YOLOv5s, YOLOv8n and YOLOv10n, and the comparison results are shown in Table 6. The comparison results show that the YOLO-MD model has the best mAP (@0.5) and the lowest parameter amount through the BDE data enhancement method, which combines the DwSPPF backbone network improvement and the GhostConv and C2fGhost neck network improvements. Compared with the original YOLOv10n model, the mAP (@0.5) of the BDE enhanced YOLO-MD model on the TMSD-3 dataset proposed in the present invention is improved by 23.7% to 87.3%. In addition, the parameter amount and GFLOPs of YOLO-MD are reduced by 347,000 and 0.9 respectively compared with YOLOv10n. Compared with the other six algorithms, BDE-enhanced YOLO-MD performs well in multiple key indicators, including average accuracy, number of parameters, amount of computation, and detection speed. This shows that the BDE-enhanced YOLO-MD algorithm not only improves detection accuracy, but also achieves effective utilization of computing resources.

[0140] Table 6

[0141] Algorithm Model mAP(@0.5) Parameter quantity / M GFLOPs FPS Faster R-CNN 47.7％ 41.76 740.9 1.59 SSD 70.5％ 24.013 90.5 10.45 YOLOv3 58.0％ 61.53 65.1 31.58 YOLOv5s 75.5％ 7.018 15.8 33.69 YOLOv8n 74％ 3.006 8.1 77.18 YOLOv10n 63.6％ 2.696 8.2 80.41 YOLO-MD(BDE) 87.3％ 2.348 7.3 100.35

[0142] In summary, the present invention demonstrates significant advantages in terms of accuracy and effective use of computing resources. It not only achieves better performance in the identification of various types of defects, but also improves the generalization ability of the model, enabling it to better adapt to defect detection tasks in complex scenarios.

[0143] This embodiment also provides a micro-nano structure defect identification device, such as Fig.13 As shown, the device comprises the following components:

[0144] The first training module 01 is used to crop the original image in the original image set of the micro-nano structure to obtain a defect sample image; and obtain a noise sample vector, and based on the defect sample image and the noise sample vector, train a deep convolutional generative adversarial network to obtain a defect image generator;

[0145] The image expansion module 02 is used to obtain a noise vector, input the noise vector into the defect image generator to obtain a virtual defect image, and fuse the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures;

[0146] The second training module 03 is used to train a defect recognition model based on the defect enhanced image set to obtain a defect recognition trained model;

[0147] The recognition module 04 is used to obtain a measured image with a micro-nano structure, and apply the defect recognition trained model to the measured image to obtain a defect recognition result.

[0148] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Fig.14 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying micro-nanostructure defects is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0149] Those skilled in the art will understand that Fig.14 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and a micro-nano structure defect recognition program stored in the memory and executable on the processor. When the processor executes the micro-nano structure defect recognition program, the following operation instructions are implemented:

[0151] The original image in the original image set of the micro-nano structure is cropped to obtain a defect sample image; and a noise sample vector is obtained, and based on the defect sample image and the noise sample vector, a deep convolutional generative adversarial network is trained to obtain a defect image generator;

[0152] Acquire a noise vector, input the noise vector into the defect image generator to obtain a virtual defect image, and fuse the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures;

[0153] Based on the defect enhanced image set, a defect recognition model is trained to obtain a defect recognition trained model;

[0154] A measured image with a micro-nano structure is acquired, and the defect recognition trained model is applied to the measured image to obtain a defect recognition result.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying micro-nanostructure defects, characterized in that: include: The original image in the original image set of the micro-nano structure is cropped to obtain a defect sample image; and obtaining a noise sample vector, and training a deep convolutional generative adversarial network based on the defect sample image and the noise sample vector to obtain a defect image generator; Acquire a noise vector, input the noise vector into the defect image generator to obtain a virtual defect image, and fuse the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures; Based on the defect enhanced image set, a defect recognition model is trained to obtain a defect recognition trained model; A measured image with a micro-nano structure is acquired, and the defect recognition trained model is applied to the measured image to obtain a defect recognition result.

2. The micro-nanostructure defect identification method according to claim 1, characterized in that: The deep convolutional generative adversarial network includes a generator and a discriminator. Based on the defect sample image and the noise sample vector, the deep convolutional generative adversarial network is trained to obtain a defect image generator, including: Input the noise sample vector into the generator to obtain a virtual sample image output by the generator; Inputting the virtual sample image and the defect sample image into the discriminator to obtain a loss function output by the discriminator, wherein the loss function is used to characterize difference information between the virtual sample image and the defect sample image; Based on the loss function, the generator and the discriminator are trained alternately, and the generator that has completed the training is used as a defect image generator.

3. The micro-nanostructure defect identification method according to claim 2, characterized in that: The number of network layers of the generator and the number of network layers of the discriminator are both related to the resolution of the defect image that the generator needs to generate.

4. The micro-nanostructure defect identification method according to claim 1, characterized in that: The virtual defect image is merged into the original image to obtain a defect enhanced image set of the micro-nano structure, including: Determine a hole defect virtual image, a stripe aggregation defect virtual image, and a nanocolumn defect virtual image in the virtual defect image; An image fusion method based on a sliding window is applied to the virtual image of the hole defect and the original image, an image fusion method based on a sliding window is applied to the virtual image of the stripe aggregation defect and the original image, and an image fusion method based on a sliding window is applied to the virtual image of the nanocolumn defect and the original image, so as to obtain a defect enhanced image set of the micro-nano structure.

5. The micro-nanostructure defect identification method according to claim 1, characterized in that: The defect recognition model is an improved YOLOv10 model, and the convolution layer in the spatial pyramid pooling module of the improved YOLOv10 model is a depth-separable convolution layer.

6. The micro-nanostructure defect identification method according to claim 5, characterized in that: The feature extraction structure in the neck of the YOLOv10 model includes a front convolution layer, a segmentation layer, a bottleneck structure group, a splicing layer, and a rear convolution layer connected in sequence, the bottleneck structure group includes several improved bottleneck structures, and each improved bottleneck structure includes a first Ghost convolution layer, a batch normalization layer, a ReLU activation function layer, a second Ghost convolution layer, and a residual connection layer connected in sequence.

7. The micro-nanostructure defect identification method according to claim 1, characterized in that: Applying the defect recognition trained model to the measured image to obtain a defect recognition result includes: The defect recognition trained model is applied to the measured image to obtain the defect type, defect size and defect position, and the defect type, defect size and defect position are used as the defect recognition result.

8. A micro-nanostructure defect identification device, characterized in that: The device comprises the following components: The first training module is used to crop the original image in the original image set of the micro-nano structure to obtain a defect sample image; and obtain a noise sample vector, and based on the defect sample image and the noise sample vector, train a deep convolutional generative adversarial network to obtain a defect image generator; An image expansion module, used for obtaining a noise vector, inputting the noise vector into the defect image generator to obtain a virtual defect image, and fusing the virtual defect image into the original image to obtain a defect enhanced image set of micro-nano structures; A second training module is used to train a defect recognition model based on the defect enhanced image set to obtain a defect recognition trained model; The recognition module is used to obtain a measured image with a micro-nano structure, and apply the defect recognition trained model to the measured image to obtain a defect recognition result.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a micro-nanostructure defect recognition program stored in the memory and executable on the processor. When the processor executes the micro-nanostructure defect recognition program, the steps of the micro-nanostructure defect recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a micro-nanostructure defect recognition program, and when the micro-nanostructure defect recognition program is executed by the processor, the steps of the micro-nanostructure defect recognition method according to any one of claims 1 to 7 are implemented.

Images