MEMS chip automatic defect classification method and system based on deep learning, and terminal

Abstract

The invention provides an MEMS chip automatic defect classification method and system based on deep learning, and a terminal, and the method comprises the steps: carrying out the preprocessing of MEMS chip image normal sample data and MEMS chip image defect sample data related to a plurality of defect classes, and constructing a training data set; and training an MEMS chip automatic defect classification model by using a comparative learning training strategy, and finally outputting a corresponding classification result according to an input MEMS chip image to be predicted by using the MEMS chip automatic defect classification model. According to the invention, the efficiency and accuracy of defect detection are improved, and various defect types can be effectively distinguished.

Description

Technical Field

[0001] The present invention relates to the field of MEMS technology, and in particular to a method, system and terminal for automatic defect classification of MEMS chips based on deep learning. Background Art

[0002] Microelectromechanical system (MEMS) chips may have a variety of defects during the manufacturing process, which will affect the performance and reliability of the chip. At present, traditional defect detection methods mainly rely on manual inspection and simple image processing technology, which have the problems of low detection efficiency and poor accuracy. With the development of deep learning technology, automated defect detection and classification methods based on neural networks have gradually become a research hotspot, but improvements are still needed to improve the accuracy and diversity of classification. Summary of the invention

[0003] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a MEMS chip automatic defect classification method, system and terminal based on deep learning, which is used to solve the technical problems that traditional defect detection methods mainly rely on manual inspection and simple image processing technology, and have low detection efficiency and poor accuracy.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for automatic defect classification of MEMS chips based on deep learning, the method comprising: normal sample data of MEMS chip images and defect sample data of MEMS chip images involving multiple defect categories; wherein the normal sample data of MEMS chip images comprises: multiple MEMS chip images respectively marked with normal labels; the defect sample data of MEMS chip images comprises: multiple MEMS chip images respectively marked with defect category labels; each MEMS chip image is preprocessed, and a training data set is constructed; based on the training data set, a deep learning network framework designed by contrastive learning training strategy is used to obtain an automatic defect classification model for MEMS chips; based on the automatic defect classification model for MEMS chips, a corresponding classification result is output according to the input MEMS chip image to be predicted; wherein the types of the classification results include: normal results and defect classification results.

[0005] In one embodiment of the present invention, the preprocessing of each MEMS chip image and the construction of a training data set include: performing image preprocessing on each MEMS chip image; converting the defect category label of each MEMS chip image into a one-hot encoding form to obtain a one-hot encoding label; and using each MEMS chip image after image preprocessing and its corresponding one-hot encoding label as a training data set.

[0006] In one embodiment of the present invention, the designed deep learning network framework includes: two deep learning network models using the same encoding-decoding structure and a classifier; wherein each encoding-decoding structure includes: an encoder, used to extract features of the image input to the model and output a feature map; a decoder, connected to the encoder, used to reconstruct the image according to the feature map output by the encoder; the classifier, connected to the encoders of the two encoding-decoding structures, used to perform feature fusion on the feature maps output by the two encoders, and output a corresponding classification result based on the feature fusion result.

[0007] In one embodiment of the present invention, a method for obtaining a MEMS chip automatic defect classification model by training a deep learning network framework designed using a contrastive learning training strategy includes: pre-training the encoding-decoding structure using the training data set; performing two different forms of data enhancement on the training data set to obtain two enhanced data sets; inputting each of the two enhanced data sets into the corresponding pre-trained encoding-decoding structure, performing contrastive learning training on the two encoders to construct two deep learning network models; and inputting the feature maps output by the two deep learning network models according to the input enhanced data sets into the classifier for training to obtain the MEMS chip automatic defect classification model.

[0008] In one embodiment of the present invention, a reconstruction loss and parameter constraints are used to pre-train the encoder-decoder structure.

[0009] In one embodiment of the present invention, the loss function used for contrastive learning training of two encoders includes: parameter constraint, feature diversity loss and contrast loss.

[0010] In one embodiment of the present invention, the loss function used in training the classifier is the cross entropy loss.

[0011] In an embodiment of the present invention, the defect categories include: bubbles, residual glue, scratches, and foreign objects.

[0012] To achieve the above-mentioned purpose and other related purposes, the present invention provides a MEMS chip automatic defect classification system based on deep learning, the system comprising: an image data acquisition module, used to acquire normal sample data of MEMS chip images and defect sample data of MEMS chip images involving multiple defect categories; wherein the normal sample data of MEMS chip images include: multiple MEMS chip images respectively marked with normal labels; the defect sample data of MEMS chip images include: multiple MEMS chip images respectively marked with defect category labels; a data preprocessing module, connected to the image data acquisition module, used to preprocess each MEMS chip image and construct a training data set; a model training module, connected to the data preprocessing module, used to obtain a MEMS chip automatic defect classification model based on the training data set and using a deep learning network framework designed by contrastive learning training strategy; a classification module, connected to the model training module, used to output corresponding classification results based on the MEMS chip automatic defect classification model according to the input MEMS chip image to be predicted; wherein the types of classification results include: normal results and defect classification results.

[0013] To achieve the above-mentioned objectives and other related objectives, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store computer programs; the one or more processors are connected to the memories and are used to run the computer programs to execute the deep learning-based MEMS chip automatic defect classification method.

[0014] As described above, the present invention is a method, system and terminal for automatic defect classification of MEMS chips based on deep learning, which has the following beneficial effects: the present invention constructs a training data set by preprocessing normal sample data of MEMS chip images and sample data of MEMS chip image defects involving multiple defect categories, and then trains the automatic defect classification model of MEMS chips using a contrastive learning training strategy, and finally uses the automatic defect classification model of MEMS chips to output the corresponding classification result according to the input MEMS chip image to be predicted. The present invention improves the efficiency and accuracy of defect detection and can effectively distinguish multiple defect types. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Shown is a flow chart of a method for automatic defect classification of MEMS chips based on deep learning in one embodiment of the present invention.

[0016] Figure 2 Shown is a schematic diagram of the structure of a deep learning network framework designed in one embodiment of the present invention.

[0017] Figure 3Shown is a schematic diagram of encoding-decoding structure pre-training in one embodiment of the present invention.

[0018] Figure 4 It is a schematic diagram of encoder contrast learning training of two encoding-decoding structures in one embodiment of the present invention.

[0019] Figure 5 Shown is a schematic diagram of classifier training in an embodiment of the present invention.

[0020] Figure 6 Shown is a schematic diagram of the complete process of training an automatic defect classification model for MEMS chips in one embodiment of the present invention.

[0021] Figure 7 Shown is a structural schematic diagram of a MEMS chip automatic defect classification system based on deep learning in one embodiment of the present invention.

[0022] Figure 8 FIG. 4 is a schematic diagram showing the structure of an electronic terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments may also be used, and that mechanical composition, structural, electrical and operational changes may be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the published patents. The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0025] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.

[0026] The terms first, second and third mentioned herein are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of the present invention.

[0027] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0028] The present invention provides a method for automatic defect classification of MEMS chips based on deep learning, which constructs a training data set by preprocessing normal sample data of MEMS chip images and sample data of MEMS chip image defects involving multiple defect categories, and then trains a MEMS chip automatic defect classification model using a contrastive learning training strategy, and finally uses the MEMS chip automatic defect classification model to output a corresponding classification result according to an input MEMS chip image to be predicted. The present invention improves the efficiency and accuracy of defect detection and can effectively distinguish multiple defect types.

[0029] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0030] like Figure 1A flowchart of a method for automatic defect classification of MEMS chips based on deep learning in an embodiment of the present invention is shown.

[0031] The method comprises:

[0032] Step S1: Acquire MEMS chip image data involving multiple defect categories.

[0033] Specifically, the normal sample data of the MEMS chip image includes: a plurality of MEMS chip images respectively marked with normal labels; the defective sample data of the MEMS chip image includes: a plurality of MEMS chip images respectively marked with defective category labels. The MEMS chip images can be manually classified and labeled by technicians.

[0034] In one embodiment, the defect categories of the MEMS chip image include:

[0035] (1) Bubbles: Circular or elliptical cavities formed on the surface or inside the MEMS chip, usually caused by incomplete gas discharge during the manufacturing process or internal defects in the material.

[0036] (2) Residual glue: Glue or adhesive that remains on the chip surface or in the gap that has not been completely removed.

[0037] (3) Scratches: Linear damage on the chip surface caused by mechanical friction or scratches.

[0038] (4) Foreign matter: Unexpected substances that appear unexpectedly on the surface or inside the chip, such as dust, particles, fibers, etc.

[0039] Step S2: Preprocess each MEMS chip image and construct a training data set.

[0040] In one embodiment, step S2 includes:

[0041] Perform image preprocessing on each MEMS chip image. Preferably, specific image preprocessing methods include grayscale, normalization, compression, etc.; grayscale converts color images into grayscale images, reducing the three-dimensional RGB space to a one-dimensional grayscale space. Color information is not necessary for defect detection and may even introduce noise. Grayscale can remove the interference of color information, significantly reduce the complexity of data processing, and make the model more focused on key features such as shape and texture. Normalization makes the data have the same scale, avoids the gradient disappearance or explosion problem caused by the difference in numerical ranges between different features, and can speed up the training of the model, so that the model converges to the optimal solution faster. Compressed image data occupies less storage space and has a faster transmission speed, which is particularly important for the processing of large-scale data sets and distributed computing.

[0042] The labels of each MEMS chip image are converted into one-hot encoding form to obtain one-hot encoding labels; specifically, the category labels are converted into numerical forms that can be processed by the machine learning model. One-hot encoding is an encoding method commonly used for classification problems. It represents each category as a binary array in which only one element is 1 (indicating the category) and the rest are 0. Assuming there are four categories (normal, bubble, residual glue, scratch), these categories can be encoded as [1,0,0,0], [0,1,0,0], [0,0,1,0] and [0,0,0,1] respectively. This encoding method eliminates the possible erroneous numerical relationships between categories and makes each category an independent feature, which helps the model better understand the differences between different categories. One-hot encoding converts the classification problem into multiple independent binary classification problems (i.e., each category corresponds to a binary bit), which simplifies the design of the model and makes it easier to extend the model to new categories.

[0043] The preprocessed image data and the corresponding one-hot encoded labels are divided into training set, validation set and test set as training data sets to evaluate the performance of the model during the training process and prevent overfitting.

[0044] Step S3: Based on the training data set, a deep learning network framework designed by contrastive learning training strategy is trained to obtain a MEMS chip automatic defect classification model.

[0045] In one embodiment, if Figure 2 , the designed deep learning network framework includes: two deep learning network models using the same encoding-decoding structure and a classifier;

[0046] Each encoding-decoding structure includes:

[0047] The encoder is used to extract features from the image input to the model and output feature maps. The encoder gradually converts the input image data into high-level feature representations, namely feature maps, through a series of convolutional layers, pooling layers, and possible nonlinear activation functions (such as ReLU). These feature maps capture key information in the image, such as edges, textures, shapes, etc., which are crucial for subsequent defect category recognition.

[0048] The decoder is connected to the encoder and is used to reconstruct the image based on the feature map output by the encoder. In this specific defect classification task, the decoder plays two important roles: first, as a regularization method, it helps the encoder learn more robust features; second, through the reconstruction process, it verifies the effectiveness of the encoder's feature extraction. The decoder usually gradually converts the feature map back to an output close to the original image size through a series of upsampling or transposed convolution layers. This process may also include convolutional layers and nonlinear activation functions to further refine the details of the reconstructed image.

[0049] The classifier connects the encoders of the two encoding-decoding structures, and is used to fuse the feature maps output by the two encoders, and output the corresponding classification results according to the feature fusion results. Feature fusion can be achieved in many ways, such as simple concatenation, weighted summation, or more complex attention mechanisms. In this framework, the feature maps output by the two encoders are fused into a comprehensive feature representation, which contains complementary information from different encoders and reduces redundancy. Based on the fused feature representation, the classifier uses a series of fully connected layers or convolutional layers, combined with appropriate nonlinear activation functions and output layers (such as Softmax layers), to output the classification results corresponding to the image.

[0050] The deep learning network framework designed in this scheme uses two parallel encoding-decoding structures to extract features from input images from different angles or levels. This multi-view feature extraction method can capture more comprehensive and rich image information, which helps the model understand the image content more accurately. During the training process, this framework not only focuses on feature extraction, but also reconstructs the image through the decoder, which helps to enhance the robustness of the model. At the same time, the combination of multi-view feature extraction and feature fusion makes the model easier to adapt to new data and scenarios, and improves the generalization ability of the model. In addition, the classifier fuses the feature maps output by the two encoders. This process can combine complementary information from different encoders, reduce redundancy, and thus enhance the recognition ability of the model. The comprehensive feature representation after feature fusion is more comprehensive and accurate, which helps to improve the accuracy of defect classification.

[0051] In one embodiment, obtaining a MEMS chip automatic defect classification model by using a deep learning network framework designed by contrastive learning training strategy training includes the following steps:

[0052] Step S31: Pre-train the encoding-decoding structure using the training data set. After the pre-training is completed, the parameters of the trained encoding-decoding structure are saved. Since the two deep learning network models use the same encoding-decoding structure, these pre-training parameters can be loaded into the corresponding parts of the two encoding-decoding structures to preliminarily construct two deep learning network models; it is also possible to only need to train one of the encoding-decoding structures, because the two encoding-decoding structures are the same, then the pre-training parameters for training one encoding-decoding structure can be loaded into the two encoding-decoding structures to initialize the next step of training.

[0053] Step S32: Perform two different forms of data enhancement on the training data set to obtain two enhanced data sets. Specifically, first, select a data enhancement technology or a combination of multiple data enhancement technologies as a form. The two forms of data enhancement may include completely different data enhancement technologies or partially different data enhancement technologies. These data enhancement technologies may be image transformation (such as rotation, flipping, scaling, cropping, color transformation, etc.), noise addition, affine transformation, mixed image (such as CutMix, MixUp), etc.

[0054] Step S33: Input the two enhanced data sets into the corresponding pre-trained encoding-decoding structure respectively, and perform contrastive learning training on the two encoders to obtain the two deep learning network models finally constructed. Specifically, one of the two enhanced data sets is input into the corresponding pre-trained encoding-decoding structure, and the remaining enhanced data set is input into another pre-trained corresponding encoding-decoding structure that is initially constructed, and the two encoders are subjected to contrastive learning training. The goal of contrastive learning is to enable the encoder to learn feature representations that are insensitive to enhanced transformations but sensitive to defect features. Save the parameters of the two trained encoders, and load the parameters into the two pre-trained corresponding encoding-decoding structures to obtain the two deep learning network models finally constructed.

[0055] Step S34: Input the feature graphs output by the two deep learning network models according to the input enhanced data sets into the classifier for training to obtain the MEMS chip automatic defect classification model. Specifically, the two encoders input their respective enhanced data sets to obtain distribution features respectively, and the feature graphs (or processed feature vectors) output by the two encoders are used as inputs of the classifier to train the classifier, obtain the final model parameters, and obtain the MEMS chip automatic defect classification model.

[0056] The training strategy of the present invention is to first use a pre-training data set to perform preliminary training on the encoder and decoder so that they have basic feature extraction and representation capabilities. When the decoder parameters remain fixed, the encoder is further trained using an enhanced data set, aiming to enhance the encoder's adaptability to specific tasks or data sets while maintaining the stability of the decoder. After the codec parameters are relatively stable, the classifier is specially trained so that the classifier can be more focused on learning how to map the features extracted by the codec to specific categories, thereby improving the accuracy of classification. During the entire training process, the parameters of the codec can also be fine-tuned to further optimize the performance of the entire network and ensure the best coordination between the various parts. The present invention can more effectively optimize network parameters by splitting the training process into multiple stages, with each stage focusing on a specific task or component, and avoiding the problem of gradient disappearance or explosion that may occur when all parts are trained at once.

[0057] In one embodiment, the encoder-decoder structure is pre-trained using reconstruction loss and parameter constraints. Figure 3 , reconstruction loss is a measure of the difference between the decoder output image and the original input image. It is often used to train the decoder part of the encoder-decoder structure to ensure that the decoder can accurately reconstruct the original image based on the feature map extracted by the encoder. The calculation of reconstruction loss is usually based on pixel-level comparison, and commonly used loss functions include mean squared error loss (MSE, Mean Squared Error). Parameter constraint is a constraint or penalty on the distribution parameters of the encoder output to ensure that these parameters conform to a certain prior distribution (such as the standard normal distribution). This constraint can be implemented through the KL divergence (Kullback-Leibler Divergence), which measures the difference between the distribution of the encoder output and the prior distribution and uses it as part of the loss function.

[0058] In one embodiment, if Figure 4 , contrast loss, diversity loss and parameter constraint are used to perform contrast learning training on the two encoders. The loss functions used in training include: parameter constraint, feature diversity loss and contrast loss.

[0059] Parameter constraints are used to fine-tune or control the parameters of the encoder during the comparison training process to ensure stability and performance during the model training process. KL divergence can be used, where KL divergence is only responsible for the parameter update of each branch.

[0060] Feature diversity loss aims to encourage the two encoders to learn different feature representations, even though they are processing the same or similar input data. This loss function is very important to avoid mode collapse and improve the generalization ability of the model. It is usually implemented by comparing the difference between the outputs of the two encoders, such as using L1 or L2 distance to measure this difference.

[0061] Contrastive loss is the core of contrastive learning, which is responsible for measuring the distance between positive pairs (i.e. similar or related samples) and negative pairs (i.e. dissimilar or unrelated samples). Contrastive loss aims to bring the distance between positive pairs closer while pushing the distance between negative pairs farther. This is usually implemented in the form of a triplet loss (Triplet Loss) or N-pair loss (N-pair Loss), but there are also some variants, such as InfoNCE loss (used in models such as CPC and SimCLR) or Normalized Temperature-scaled Cross-Entropy Loss (NT-Xent, used in SimCLR).

[0062] Assume that the total loss function is L total , the KL divergence loss is L KL , the feature diversity loss is L diversity , the contrast loss is L contrast , then the total loss function can be expressed as:

[0063] L total =αL KL +βL diversity +γL contrast ; (1)

[0064] Among them, α, β, and γ are hyperparameters used to control the contribution of each loss function to the total loss. The choice of these hyperparameters depends on the specific task, dataset, and model architecture.

[0065] In one embodiment, if Figure 5, the loss function used to train the classifier is the cross-entropy loss. Specifically, in the process of training the classifier, the cross-entropy loss function is used as the benchmark for evaluation and optimization. Cross-entropy loss is a commonly used loss function, especially when dealing with classification problems. It measures the difference between two probability distributions, especially the difference between the output predicted by the classifier (i.e., the predicted probability distribution) and the true label (i.e., the actual probability distribution, usually one-hot encoding), providing a clear measure of model performance. Especially when dealing with multi-classification tasks, cross-entropy loss is known for its high sensitivity to prediction accuracy and can effectively capture the mismatch between the predicted probability distribution and the true distribution. This feature prompts the model to continuously adjust its parameters in training iterations to minimize the cross-entropy loss, thereby optimizing classification performance and improving prediction accuracy.

[0066] After training using reconstruction loss, parameter constraints, contrast loss, diversity loss, and cross entropy loss, the final encoder and classifier parameters are obtained, and the parameters are loaded into the deep learning network framework to build the MEMS chip automatic defect classification model. Figure 6 shown.

[0067] In one embodiment, the classifier includes two parts: feature fusion and classification; the feature fusion part is used to fuse the feature maps output by the two encoders. The classification part is used to output the corresponding classification result according to the feature fusion result. In the process of feature fusion, if the feature maps output by the two encoders are too similar, the fused features may not be able to fully utilize the advantages of the two encoders, resulting in information redundancy and performance bottlenecks. Therefore, in addition to the cross entropy loss, the feature diversity loss can also be used to train the feature fusion part before classification after feature fusion. By introducing the feature diversity loss, the model can be prompted to learn more unique and complementary features, thereby improving the performance of the classifier.

[0068] Step S4: Based on the MEMS chip automatic defect classification model, a corresponding classification result is output according to the input MEMS chip image to be predicted.

[0069] Specifically, the MEMS chip image to be predicted is input into a trained MEMS chip automatic defect classification model, and the corresponding classification result is output, wherein the types of the classification result include: normal result and defect classification result.

[0070] Similar to the principle of the above embodiment, the present invention provides a MEMS chip automatic defect classification system based on deep learning.

[0071] The following provides specific embodiments in conjunction with the accompanying drawings:

[0072] like Figure 7 A structural schematic diagram of a MEMS chip automatic defect classification system based on deep learning in an embodiment of the present invention is shown.

[0073] The system comprises:

[0074] Image data acquisition module 1, used for MEMS chip image normal sample data and MEMS chip defect image data involving multiple defect categories; wherein the MEMS chip image normal sample data includes: multiple MEMS chip images respectively marked with normal labels; the MEMS chip image defect sample data includes: multiple MEMS chip images respectively marked with defect category labels;

[0075] A data preprocessing module 2, connected to the image data acquisition module 1, is used to preprocess the images of each MEMS chip and construct a training data set;

[0076] A model training module 3, connected to the data preprocessing module 2, is used to obtain a MEMS chip automatic defect classification model based on a training data set and a deep learning network framework designed by a contrastive learning training strategy;

[0077] The classification module 4 is connected to the model training module 3 and is used to output corresponding classification results based on the input MEMS chip image to be predicted based on the MEMS chip automatic defect classification model; wherein the types of the classification results include: normal results and defect classification results.

[0078] It should be understood that Figure 7 The division of each module in the system embodiment is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And these units can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of processing elements calling software, and some units can be implemented in the form of hardware.

[0079] Since the implementation principle of the deep learning-based MEMS chip automatic defect classification system has been described in the previous embodiments, it will not be repeated here.

[0080] The MEMS chip automatic defect classification method based on deep learning provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 8, is an optional hardware structure diagram of a deep learning-based MEMS chip automatic defect classification terminal 1000 provided in an embodiment of the present invention. The terminal 1000 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 8 In the specification, various buses are labeled as bus systems.

[0081] The user interface 1009 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0082] It is understood that the memory 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0083] The memory 1002 in the embodiment of the present invention is used to store various categories of data to support the operation of the terminal 1000. Examples of these data include: any executable program for operating on the terminal 1000, such as an operating system 10021 and an application 10022; the operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 10022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The deep learning-based MEMS chip automatic defect classification method provided in the embodiment of the present invention may be included in the application 10022.

[0084] The method disclosed in the above embodiment of the present invention can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1001 or the instruction in the form of software. The above processor 1001 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 1001 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0085] In an exemplary embodiment, the terminal 1000 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0086] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0087] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0088] In summary, the MEMS chip automatic defect classification method, system and terminal based on deep learning of the present invention constructs a training data set by preprocessing normal sample data of MEMS chip images and defect sample data of MEMS chip images involving multiple defect categories, and then uses a comparative learning training strategy to train the MEMS chip automatic defect classification model. Finally, the MEMS chip automatic defect classification model is used to output the corresponding classification result according to the input MEMS chip image to be predicted. The present invention improves the efficiency and accuracy of defect detection and can effectively distinguish multiple defect types. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0089] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for automatic defect classification of MEMS chips based on deep learning, characterized in that: The method comprises: Acquire normal sample data of MEMS chip images and defective sample data of MEMS chip images involving multiple defect categories; wherein the normal sample data of MEMS chip images includes: multiple MEMS chip images respectively marked with normal labels; the defective sample data of MEMS chip images includes: multiple MEMS chip images respectively marked with defective category labels; Preprocess each MEMS chip image and construct a training data set; Based on the training data set, the deep learning network framework designed by contrastive learning training strategy is used to obtain the MEMS chip automatic defect classification model; Based on the MEMS chip automatic defect classification model, a corresponding classification result is output according to the input MEMS chip image to be predicted; wherein the types of the classification result include: a normal result and a defect classification result.

2. The method for automatic defect classification of MEMS chips based on deep learning according to claim 1, characterized in that: The preprocessing of each MEMS chip image and constructing a training data set includes: Perform image preprocessing on each MEMS chip image; Convert the label of each MEMS chip image into a one-hot encoding form to obtain a one-hot encoding label; The MEMS chip images after image preprocessing and their corresponding one-hot encoding labels are used as training data sets.

3. The method for automatic defect classification of MEMS chips based on deep learning according to claim 1, characterized in that: The designed deep learning network framework includes: two deep learning network models using the same encoding-decoding structure and a classifier; Among them, each encoding-decoding structure includes: The encoder is used to extract features from the image input to the model and output a feature map; A decoder, connected to the encoder, configured to reconstruct an image according to a feature map output by the encoder; The classifier is connected to the encoders of the two encoding-decoding structures, and is used to perform feature fusion on the feature maps output by the two encoders, and output a corresponding classification result according to the feature fusion result.

4. The method for automatic defect classification of MEMS chips based on deep learning according to claim 3, characterized in that: The methods of using the contrastive learning training strategy to train the designed deep learning network framework to obtain the MEMS chip automatic defect classification model include: Pre-training the encoding-decoding structure using the training data set; Performing two different forms of data augmentation on the training data set to obtain two augmented data sets; The two augmented datasets are respectively input into the corresponding pre-trained encoding-decoding structure, and the two encoders are trained by contrastive learning to construct two deep learning network models; The feature maps output by the two deep learning network models according to the input enhanced data set are input into the classifier for training to obtain the MEMS chip automatic defect classification model.

5. The method for automatic defect classification of MEMS chips based on deep learning according to claim 4, characterized in that: Pre-train the encoder-decoder structure using reconstruction loss and parameter constraints.

6. The method for automatic defect classification of MEMS chips based on deep learning according to claim 4, characterized in that: The loss functions used for contrastive learning training of the two encoders include: parameter constraint, feature diversity loss and contrastive loss.

7. The method for automatic defect classification of MEMS chips based on deep learning according to claim 4, characterized in that: The loss function used to train the classifier is cross entropy loss.

8. The method for automatic defect classification of MEMS chips based on deep learning according to claim 1, characterized in that: The defect categories include: bubbles, residual glue, scratches and foreign objects.

9. A MEMS chip automatic defect classification system based on deep learning, characterized in that: The system comprises: An image data acquisition module is used to acquire normal sample data of MEMS chip images and defective sample data of MEMS chip images involving multiple defect categories; wherein the normal sample data of MEMS chip images includes: multiple MEMS chip images respectively marked with normal labels; the defective sample data of MEMS chip images includes: multiple MEMS chip images respectively marked with defect category labels; A data preprocessing module, connected to the image data acquisition module, for preprocessing each MEMS chip image and constructing a training data set; A model training module, connected to the data preprocessing module, is used to obtain a MEMS chip automatic defect classification model based on a training data set and a deep learning network framework designed by contrastive learning training strategy training; A classification module is connected to the model training module and is used to output corresponding classification results based on the input MEMS chip image to be predicted based on the MEMS chip automatic defect classification model; wherein the types of the classification results include: normal results and defect classification results.

10. An electronic terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors, connected to the memory, are configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images