Data classification method and device, equipment, storage medium and program product

Abstract

The invention discloses a data classification method and device, equipment, a storage medium and a program product, and relates to the technical field of data classification, and the method comprises the steps: determining the complexity corresponding to to-be-classified picture data; the to-be-classified picture data are picture data of target objects with defects in the wafer production process; adjusting a pre-trained initial classification model based on the model structure information adaptive to the complexity; and performing classification prediction on to-be-classified picture data by using the adjusted classification model. In this way, the complexity of the to-be-classified picture data can be combined, the model structure information suitable for the to-be-classified picture data is determined from the initial classification model, and the adjusted classification model corresponding to the model structure information is utilized to perform classification prediction on the to-be-classified picture data; the initial classification model can be dynamically adjusted in real time according to different to-be-classified picture data, and an adjusted classification model adapted to each classification prediction is obtained; the model adaptability can be improved, and the classification efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data classification, and in particular to a data classification method, device, equipment, storage medium and program product. Background Art

[0002] When classifying multiple defect types using common measurement equipment at present, it is first necessary to collect a large amount of data of different defect categories and perform annotation. Based on the collected data, users manually continuously select and adjust appropriate attributes and classification rules, or continuously adjust model parameters and structures. Finally, a suitable model is output, and the classification result is obtained according to the trained model. On the one hand, when dealing with classification problems, in the face of minority class samples, it is difficult for the model to capture the defect attribute characteristics of this category, and it is easy to produce prediction results biased towards the majority class. Thus, it affects the performance of the overall classification result. If it is necessary to balance the samples of all defect categories, a large amount of manpower is required to annotate the data. On the other hand, defect samples of different categories may have different characteristics and reasoning difficulties. For simple samples, the deep network may perform excessive calculations, resulting in waste of resources and low prediction efficiency; while for complex samples, a deeper network may be required to capture features. When performing defect classification based on the trained model, the fixed structure often cannot effectively cope with the diversity and complexity of the input data, resulting in a decrease in classification accuracy.

[0003] Therefore, how to use a suitable network model to classify and predict data is a problem to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data classification method, device, equipment, storage medium and program product, which is used to solve the adaptability problem of the model for data classification. The specific solutions are as follows:

[0005] In a first aspect, the present application provides a data classification method, including: determining the complexity corresponding to the picture data to be classified; the picture data to be classified is the picture data of the target object with defects in the wafer production process; adjusting a pre-trained initial classification model based on the model structure information adapted to the complexity; using the adjusted classification model to perform classification prediction on the picture data to be classified. In this solution, the model structure information suitable for the picture data to be classified can be determined from the initial classification model in combination with the complexity of the picture data to be classified, and the adjusted classification model corresponding to the model structure information is used to perform classification prediction on the picture data to be classified; the initial classification model can be dynamically adjusted in real time according to the differences of the picture data to be classified, and the adjusted classification model adapted to each classification prediction can be obtained; in this way, the model adaptability can be improved and the classification efficiency can be improved.

[0006] In one possible implementation, determining the complexity corresponding to the image data to be classified includes: evaluating the complexity corresponding to the image data to be classified using at least one complexity evaluator in the initial classification model; wherein different complexity evaluators are pre-installed at different jump connections in the initial classification model. In this implementation, complexity evaluators are installed at different jump connections in the initial classification model, enabling real-time complexity evaluation of input data within the model to facilitate timely adjustment of the corresponding model structure.

[0007] In one possible embodiment, evaluating the complexity of the image data to be classified includes: extracting features from a target defect image block using the complexity evaluator at the jump connection, and performing a complexity evaluation on the extracted features; wherein the target defect image block is a defect image block corresponding to the image data to be classified received at the jump connection. In this embodiment, the complexity evaluator can extract features from the defect image block received at the jump connection, and then perform a complexity evaluation on the extracted features. This allows for real-time processing of data received at the jump connection of the model, as well as real-time complexity evaluation of the data received at the jump connection, to facilitate timely adjustment of the model structure.

[0008] In one possible embodiment, performing complexity assessment on the extracted features includes: processing the extracted features using a multilayer perceptron; and mapping the output of the multilayer perceptron to a preset complexity range using a preset activation function to obtain the complexity corresponding to the extracted features. In this embodiment, the extracted features can be processed using a multilayer perceptron and associated activation functions to learn the complex features of the data, and the output can be mapped to a preset complexity range to obtain the corresponding complexity; this can improve the credibility and efficiency of the complexity assessment.

[0009] In one possible embodiment, performing complexity assessment on the extracted features includes: determining corresponding target features from the extracted features based on a preset complexity evaluation index; and performing complexity assessment on the target features; wherein the extracted features include any one or more of defect size information, energy focusing, area, aspect ratio, noise information, and resolution corresponding to the target defect image block. In this embodiment, during the complexity assessment process, several features of the target defect image block can be freely selected for evaluation, which can improve the flexibility and scenario adaptability of the complexity assessment.

[0010] In a possible implementation manner, the model structure information includes any one or several of model output position information, model layer information, and model neuron information. In this implementation manner, according to the complexity corresponding to the actual data, several combinations of model output position information, model layer information, and model neuron information can be freely selected, so that the model structure can be flexibly adjusted to improve the accuracy of data classification.

[0011] In a possible implementation manner, adjusting the pre-trained initial classification model based on the model structure information adapted to the complexity includes: obtaining the scene information of the first picture acquisition scene; the first picture acquisition scene is the picture acquisition scene corresponding to the picture data to be classified; adjusting the pre-trained initial classification model based on the model structure information adapted to the scene information and the complexity. In this implementation manner, during the process of adjusting the model structure, the first picture acquisition scene corresponding to the picture data to be classified can be considered, and the model structure of the initial classification model can be adjusted based on the corresponding scene information and the complexity of the picture data to be classified, which can ensure the matching degree between the adjusted model and the picture data to be classified and further improve the final data classification prediction effect.

[0012] In a possible implementation manner, the method may further include: obtaining a training data set collected in the second picture acquisition scene; training a model based on the training data set to obtain the initial classification model; where the second picture acquisition scene and the first picture acquisition scene are the same scene or different scenes. In this implementation manner, during the process of training the initial classification model, the second picture acquisition scene corresponding to the training data set used can be the same as the first picture acquisition scene corresponding to the picture data to be classified, or any other different scene, which can make it easier to train the initial classification model.

[0013] In a possible implementation manner, during the training process of the initial classification model, it includes: controlling the model convergence based on the cross-entropy loss function, and dynamically adjusting the learning rate of each model parameter based on the adaptive moment estimation algorithm. In this implementation manner, during the training process of the initial classification model, the cross-entropy loss function can be used to accelerate the training process, and the learning rate of the model parameters can be dynamically adjusted based on the adaptive moment estimation algorithm, making the model training more stable and efficient.

[0014] In a possible implementation, after classifying and predicting the to-be-classified picture data by using the adjusted classification model, the method further includes: transmitting target data content including the current classification prediction result to a preset interaction interface; obtaining user feedback data for the target data content on the preset interaction interface; and performing model retraining by using a new training data set generated based on the user feedback data. In this implementation, after classifying and predicting the to-be-classified picture data by using the adjusted model, relevant data content of the classification prediction can be displayed, and a new training data set can be generated based on the user feedback data for model retraining; in this way, the classification effect of the model can be manually judged in a timely manner, the model can be optimized, and the classification prediction ability can be gradually improved.

[0015] In a possible implementation, performing model retraining by using the new training data set generated based on the user feedback data includes: performing retraining on the initial classification model or the adjusted classification model by using the new training data set. In this implementation, the model retraining process can be to retrain the initial classification model or to retrain the adjusted classification model; after each data classification, it is possible to flexibly select to retrain the entire initial classification model or only train the network related to the adjusted classification model according to actual needs to further optimize the model.

[0016] In a second aspect, the present application provides a data classification device, including:

[0017] A complexity determination module, configured to determine the complexity corresponding to the to-be-classified picture data; the to-be-classified picture data is picture data of a target object with defects in the wafer production process;

[0018] A model adjustment module, configured to adjust a pre-trained initial classification model based on model structure information adapted to the complexity;

[0019] A classification prediction module, configured to classify and predict the to-be-classified picture data by using the adjusted classification model.

[0020] In a third aspect, the present application provides an electronic device, including:

[0021] A memory, configured to store a computer program;

[0022] A processor, configured to execute the computer program to implement the data classification method as described above.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, and when the computer program is executed by a processor, the data classification method as described above is implemented.

[0024] Fifth aspect, the present application provides a computer program product, including computer programs / instructions, which when executed by a processor implement the data classification method as described above.

[0025] It can be seen that the present application first determines the complexity corresponding to the picture data to be classified; the picture data to be classified is the picture data of the target object with design defects in the wafer production process; then, based on the model structure information adapted to the complexity, the pre-trained initial classification model is adjusted; and then the adjusted classification model is used to classify and predict the picture data to be classified. In this way, the present application can combine the complexity of the picture data to be classified to determine the model structure information suitable for the picture data to be classified from the pre-trained initial classification model, and then adjust the initial classification model to obtain the adjusted classification model corresponding to the model structure information, and use the adjusted classification model to classify and predict the picture data to be classified; during the process, there is no need to re-train or pre-train multiple classification models, and the appropriate model structure can be selected in real time according to the differences of the picture data to be classified, and then the initial classification model can be dynamically adjusted according to the real-time model structure to obtain the adjusted classification model adapted to each classification prediction; in this way, the adaptability of the classification model is improved, and the prediction efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings according to the provided drawings without creative efforts.

[0027] Figure 1 It is a schematic structural diagram of a data classification system disclosed in the present application;

[0028] Figure 2 It is a flowchart of a data classification method disclosed in the present application;

[0029] Figure 3 It is a flowchart of a specific data classification method disclosed in the present application;

[0030] Figure 4 It is a flowchart of another specific data classification method disclosed in the present application;

[0031] Figure 5 It is a flowchart of a specific complexity evaluation disclosed in the present application;

[0032] Figure 6 It is a flowchart of a specific model adjustment disclosed in the present application;

[0033] Figure 7 Another specific model adjustment flowchart disclosed in this application;

[0034] Figure 8 A specific model iteration and optimization flowchart disclosed in this application;

[0035] Figure 9 A schematic structural diagram of a data classification device disclosed in this application;

[0036] Figure 10 A structural diagram of an electronic device disclosed in this application. Detailed implementation manners

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0038] In the currently commonly used data classification and prediction solutions, the first is the rule-based classification algorithm. For each piece of defective data to be classified, the user observes and analyzes the attribute characteristics of the detected defects, and continuously formulates and adjusts different classification rules and corresponding defect categories according to the attribute distribution. Finally, the defects that meet the rules will be classified into the corresponding categories. This requires manually formulating appropriate classification rules, resulting in low efficiency. The second is the classification algorithm based on deep learning, which can label the collected defective data of different categories. According to the defective data of different categories, the model is designed, including the number of layers, the number of nodes, the loss function, and the optimizer, etc. After the model is trained, during prediction, a fixed model is imported, and the subsequent new input samples are classified and predicted through this fixed model. Since the pre-trained model is fixed during subsequent classification and prediction, its adaptability to different scenarios is low. This model only has good classification and prediction capabilities for the scenarios corresponding to the samples used during model training; the model lacks generalization ability and cannot adapt to defect classification tasks under different lighting conditions and different collection modes. The third is the classification method based on transfer learning. Based on the defective data in the existing scenarios, a suitable model is trained; for the defective data in the new scenarios to be classified subsequently, or new defect categories, fine-tuning training needs to be performed based on the existing model, so as to obtain a model adapted to the new scenarios or new defect categories. In this way, for different data to be classified, continuous adjustment may be required, increasing the additional model training time, and the adjusted model is also fixed, and the prediction efficiency cannot be optimized according to different input samples during classification and prediction.

[0039] In this application, the system structure for the complexity evaluation and classification of sample data is as follows Figure 1 shown, which consists of a sample complexity evaluator (201), an adaptive dynamic classifier (202), and an online feedback module (203). Among them, instructions are input through the host computer (101), data input, complexity evaluation, and data classification are provided through the cluster (102), and the interactive interface (103) displays the sample complexity results of the sample complexity evaluator, the model results of the adaptive dynamic classifier, and provides high-definition defect pictures and model results for user online feedback. In this way, the model structure information suitable for the picture data to be classified can be determined from the pre-trained initial classification model by combining the complexity of the picture data to be classified. Subsequently, the initial classification model is adjusted to obtain the adjusted classification model corresponding to the model structure information, and the adjusted classification model is used to classify and predict the picture data to be classified; during the process, there is no need to retrain or pre-train multiple classification models, and the appropriate model structure can be selected in real time according to the differences of the picture data to be classified, and then the initial classification model is dynamically adjusted according to the real-time model structure to obtain the adjusted classification model suitable for each classification prediction; in this way, the adaptability of the classification model is improved, and the prediction efficiency can be improved. It can be understood that the technical solution of this application can be used in various defect detection devices, such as bright-field patterned wafer detection devices, dark-field patterned wafer detection devices, dark-field unpatterned wafer detection devices, etc.

[0040] Embodiment 1

[0041] See Figure 2 shown, the embodiment of the present invention discloses a data classification method, including:

[0042] Step S11, determining the complexity corresponding to the picture data to be classified; the picture data to be classified is the picture data of the target object with defects in the wafer production process.

[0043] In the embodiment of this application, the input picture data to be classified is the picture data of the target object with defects due to design, process, or external influences during the wafer generation process, such as the picture data of the wafer or mask with defects; during the data classification process, it is first necessary to determine the complexity corresponding to the input picture data to be classified, so as to determine the model structure suitable for data classification according to the complexity.

[0044] In a specific embodiment, the determination of the complexity corresponding to the picture data to be classified may include: evaluating the complexity corresponding to the picture data to be classified through at least one complexity evaluator in the initial classification model; wherein different complexity evaluators are respectively preset at different skip connections in the initial classification model. Specifically, the complexity evaluators are set at different skip connections in the initial classification model, and can evaluate the complexity of the data input to this skip connection, so as to adjust the model structure at this skip connection based on the evaluated complexity, which can flexibly adjust the model structure.

[0045] In a specific embodiment, the evaluation of the complexity corresponding to the picture data to be classified may include: extracting features from the target defect image block through the complexity evaluator at the skip connection, and evaluating the complexity of the extracted features; wherein the target defect image block is the defect image block corresponding to the picture data to be classified received at the skip connection. Specifically, in the process of evaluating the complexity of relevant data through the complexity evaluator at the skip connection, the data received at the skip connection can be the defect image block corresponding to the picture data to be classified; it can be understood that these defect image blocks are the image blocks obtained by processing the picture data to be classified through a detection algorithm, or the image blocks processed by the previous network layer corresponding to this skip connection. The complexity evaluator can extract features from the target defect image block and evaluate the complexity of the extracted features; and the corresponding complexity evaluation result can be obtained.

[0046] In another specific embodiment, the evaluation of the complexity of the extracted features may include: processing the extracted features based on a multi-layer perceptron; using a preset activation function to map the output result of the multi-layer perceptron to a preset complexity value range to obtain the complexity corresponding to the extracted features. Specifically, in the process of evaluating the complexity of the features extracted from the target defect image block, it can be processed by a multi-layer perceptron and related activation functions. The multi-layer perceptron can learn the complex features of the image block through a series of transformations and non-linear activation functions and generate an output result of complexity; then the preset activation function maps this output result to the preset complexity value range (such as 0 to 1), and the final complexity evaluation result can be obtained.

[0047] In yet another specific embodiment, the complexity evaluation of the extracted features may include: determining corresponding target features from the extracted features based on a preset complexity evaluation index; evaluating the complexity of the target features; wherein the extracted features include any one or more of the defect size information, energy focusing condition, area, aspect ratio, noise information, and resolution corresponding to the target defect image block. Specifically, the complexity evaluation process requires the use of a preset complexity evaluation index, which characterizes which parameters are used for complexity evaluation, such as any one or more of defect size information, energy focusing condition, area, aspect ratio, noise information, and resolution; in this way, the complexity evaluation index can be flexibly set to optimize the corresponding complexity evaluation effect; based on this preset complexity evaluation index, the target features for evaluating complexity can be determined from the extracted features corresponding to the image block, and then the complexity of these target features is evaluated to obtain the corresponding complexity evaluation result.

[0048] Step S12: Adjust the pre-trained initial classification model based on the model structure information adapted to the complexity.

[0049] In the embodiment of the present application, the complexity corresponding to the to-be-classified picture data can be obtained through the above steps, and then the pre-trained initial classification model can be adjusted based on this complexity; it can be understood that the model structure information corresponding to different complexities is also different, and the specific model structure involved can include any one or more of the model output position information, model layer information, and model neuron information; for example, if the complexity of the data is relatively low, the number of network layers used can be reduced, or the number of neurons can be reduced, thereby reducing the computational amount and improving the classification and inference speed of the model.

[0050] Step S13: Use the adjusted classification model to perform classification prediction on the to-be-classified picture data.

[0051] In the embodiment of the present application, the initial classification model can be adjusted based on the model structure information adapted to the complexity of the to-be-classified picture data through the above steps to obtain the adjusted classification model. It should be noted that in a specific embodiment, the adjusted classification model can be a model directly adjusted on the initial classification model or a model adjusted using a copy of the initial classification model. Further, after obtaining the adjusted classification model, the to-be-classified picture data can be classified and predicted to obtain the corresponding classification prediction result.

[0052] In a specific embodiment, after using the adjusted classification model to perform classification prediction on the image data to be classified, the method may further include: transmitting the target data content including the current classification prediction result to a preset interactive interface; obtaining user feedback data regarding the target data content on the preset interactive interface; and retraining the model using a new training dataset generated based on the user feedback data. Specifically, after the classification prediction, the corresponding current classification prediction result and the previously input (target) data content, such as the image data to be classified, may be transmitted to a preset interactive interface, through which the classification prediction result may be displayed to the relevant user; and user feedback data regarding the target data content input by the user may be obtained through the interactive interface. It is understood that this user feedback data represents the manual judgment related to the classification prediction result and the adjustment opinions related to the model; based on this user feedback data, a new training dataset may be generated, and this new training dataset may be used to perform model retraining operations to train and tune the model.

[0053] In another specific embodiment, the model retraining using a new training dataset generated based on the user feedback data may include: retraining the initial classification model or the adjusted classification model using the new training dataset. It is understood that if the initial classification model itself is adjusted based on the model structure information corresponding to the complexity, that is, the adjusted model represents the connection relationship between certain network layers and neurons in the initial classification model and is an abstract model, then when the model is retrained using the new training dataset, the initial classification model is trained and the relevant model parameters are adjusted; then, when faced with data of the same complexity, the model is re-adjusted based on the model structure information before classification prediction is performed. Correspondingly, if a copy of the initial classification model is adjusted based on the model structure information corresponding to the complexity, that is, the adjusted model is the actual model obtained by adjusting the copy of the initial classification model, and there are two actual models, the initial classification model and the adjusted model, then the model retraining using the new training dataset is actually training the actual adjusted model and optimizing the adjusted model so that when faced with data of the same complexity, the retrained adjusted model can be directly used for classification prediction.

[0054] It can be seen that the present application can determine the model structure information suitable for the to-be-classified image data from the pre-trained initial classification model in combination with the complexity of the to-be-classified image data, and then adjust the initial classification model to obtain an adjusted classification model corresponding to the model structure information, and use the adjusted classification model to perform classification prediction on the to-be-classified image data; during the process, it is not necessary to re-train or pre-train multiple classification models, and the appropriate model structure can be selected in real time according to the differences of the to-be-classified image data, and then the initial classification model can be dynamically adjusted according to the real-time model structure to obtain an adjusted classification model adapted to each classification prediction; in this way, the depth and width of the network can be flexibly adjusted, thereby improving the computing efficiency and inference performance of the network, while maintaining the adaptability and robustness to different defect data.

[0055] Embodiment 2

[0056] Based on the content disclosed in the above embodiments, refer to Figure 3 As shown, this embodiment discloses a data classification method, including:

[0057] Step S21, determine the complexity corresponding to the to-be-classified image data; the to-be-classified image data is the image data of the target object with defects in the wafer production process.

[0058] Step S22, obtain the scene information of the first image acquisition scene; the first image acquisition scene is the image acquisition scene corresponding to the to-be-classified image data.

[0059] In the embodiment of the present application, the image acquisition scene corresponding to the obtained to-be-classified image data is the first image acquisition scene, and the scene information of the classification prediction also needs to be considered during the classification prediction; the image data collected in different scenes also has an impact on the adapted model structure and the classification prediction effect.

[0060] It should be noted that, in a specific embodiment, it may further include: obtaining a training data set collected in a second image acquisition scene; performing model training based on the training data set to obtain the initial classification model; wherein, the second image acquisition scene and the first image acquisition scene are the same scene or different scenes. Specifically, during the training process of the initial classification model, the training data set used is the image data collected in the second image acquisition scene; the second image acquisition scene may be the same scene as the first image acquisition scene of the to-be-classified image data, or any other scene; during the process of data classification, it is necessary to adjust the model structure of the initial classification model according to the to-be-classified data, and the finally used adjusted model is matched with the to-be-classified data, that is, there is no necessary connection between the acquisition scene of the training data set of the initial classification model and the acquisition scene of the data for subsequent data classification.

[0061] Step S23: Adjust the pre-trained initial classification model based on the scene information and the model structure information adapted to the complexity.

[0062] In this embodiment, the complexity of the image data to be classified and the scene information of the first image acquisition scene corresponding to the image data to be classified can be obtained through the above steps; in order to ensure that the classification model used subsequently matches the image data to be classified, appropriate model structure information can be determined based on the scene information and complexity, and then the pre-trained initial classification model can be adjusted using the model structure information, so that the corresponding adjusted model is adapted to the image data to be classified, so that the adjusted model has a better prediction effect on the image data to be classified.

[0063] Step S24: Use the adjusted classification model to perform classification prediction on the image data to be classified.

[0064] For more specific processing procedures of the above steps S21 and S24, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.

[0065] It can be seen that the present application can combine the collection scenario and complexity of the image data to be classified, determine the model structure information suitable for the image data to be classified from the pre-trained initial classification model, and then adjust the initial classification model to obtain the adjusted classification model corresponding to the model structure information, and use the adjusted classification model to perform classification prediction on the image data to be classified, thereby reducing the risk of overfitting of the model on simple samples; there is no need to retrain or pre-train multiple classification models in the process, and the appropriate model structure can be selected in real time according to the different image data to be classified, and then the initial classification model can be dynamically adjusted according to the real-time model structure to obtain the adjusted classification model that adapts to each classification prediction; in this way, the depth and width of the network can be flexibly adjusted, thereby improving the computational efficiency and reasoning performance of the network, while maintaining adaptability and robustness to different defect data.

[0066] Embodiment 3

[0067] Based on the contents disclosed in the above embodiments, see Figure 4 As shown, this embodiment discloses a data classification method, which specifically includes:

[0068] In this embodiment, defect data to be classified can be input to the host computer, such as data on wafers or masks with design defects. This data includes specific defect images, defect attributes derived from the detection algorithm, and defect patches (image blocks). Accordingly, the optical scene information corresponding to the defect data also needs to be pre-configured. The defect detection equipment can be used to set parameters such as the collection mode, lighting mode, and power to perform classification and prediction of the defect data in the corresponding scenario.

[0069] Furthermore, a model file corresponding to a pre-trained initial classification model is preset in the defect detection device. The scenario corresponding to the dataset used during the training of the initial classification model has no direct relation to the scenario information corresponding to the defect data for which classification prediction is currently required. Subsequently, the initial classification model can be adjusted according to the defect data, the set optical scenario information, and the complexity of the defect data. It should be noted that during the process of evaluating the complexity of the defect data, a complexity evaluator is set at each skip connection of the model network to evaluate the complexity of the defect data, so as to adjust the model structure at the skip connection. Specifically, during the process of evaluating the complexity of the defect data, when the input data passes through a skip connection, the complexity evaluator can extract the features of the data and dynamically evaluate its complexity, thereby determining whether it is necessary to adjust the number of network layers and the number of neurons in each layer. This mechanism enables the network to flexibly adjust the use of computing resources according to the actual complexity of the input data, avoid overly deep or shallow network structures, and improve the computing efficiency and inference performance. In a specific embodiment, as Figure 5 shown in the complexity evaluation process, the complexity evaluation module can be composed of two main parts: a feature extraction module and a decision unit module. The feature extraction module is responsible for extracting meaningful features from the defect Patch map corresponding to the input defect data, and these features can reflect the complexity of the sample. The extracted features will be passed to the decision unit, and the decision unit evaluates the complexity based on these features and outputs a decision value indicating the high or low complexity of the sample. This decision value determines whether to skip certain layers or adjust the number of neurons to meet the processing requirements of the current sample. In network design, the decision unit usually consists of a shallow Multilayer Perceptron (MLP) and a Sigmoid function (an activation function used for binary classification tasks and the output layer). The MLP learns the complex features of the input sample through a series of linear transformations and non-linear activation functions and generates a complexity score. The Sigmoid activation function maps the output value to between 0 and 1 as the final decision value. If the sample complexity is low, the decision unit will instruct the network to skip certain layers or reduce the number of neurons in the current layer, thereby reducing the computational amount and improving the inference speed.

[0070] It should be noted that during the training process of the initial classification model, in order to enable the system to flexibly select a suitable defect classification network model for classification according to the characteristics of the input defect data, that is, to adaptively adjust the model structure for defect classification; during the initialization stage of the model, it can be trained based on a large number of existing labeled defect category image data. Specifically, the data used to train the model can include defect images collected from different scenarios such as different lighting conditions, collection modes, power, etc., covering various types of defects. Through preliminary training with a deep adaptive network, during the training process, the cross-entropy loss function can be used as the optimization objective. This is a loss function commonly used in classification tasks and can effectively measure the difference between the prediction result and the actual label. To accelerate the training process and improve the convergence speed of the model, the Adam optimizer (Adaptive Moment Estimation) can be adopted. This optimizer can dynamically adjust the learning rate of each parameter, making the model training process more efficient and stable. The adaptive mechanism is one of the core features of the data classification process. The goal of this mechanism is to select the most suitable network model for classification according to the different characteristics of the input defect data, such as the size, signal-to-noise ratio, area, etc. of the defect, during the model prediction stage. Specifically, as Figure 6 shows the process of adjusting the model structure according to complexity. When the model processes data, it will dynamically adjust its structure based on the following aspects: Dynamically determine the result output: The network can dynamically determine where to output the final prediction result according to the specific characteristics of the input data. For example, when the resolution of the image of the input defect sample is low, the network can choose to terminate the prediction after a shallow structure and output the result to reduce the computational complexity; while when the image of the defect sample contains more details and complex features, the network can automatically choose to enter a deeper network to capture more abundant information and then output the predicted classification result. Dynamically adjust the network levels: The adjustment of the network levels means that the network can flexibly adjust the number of layers when processing tasks of different complexities. For example, when dealing with relatively simple and common defect types, the system may only need fewer convolutional layers and fully connected layers, while for the classification of complex defects, the network may require more levels of convolution and deep feature extraction. This dynamic adjustment can significantly improve the computational efficiency of the model and ensure that it can handle various complex situations. Dynamically delete neurons: The number and activation degree of neurons in the network will be dynamically adjusted according to the characteristics of the input data. When the quality of the input defect data is poor or there is more noise, the system may choose to disable some neurons to reduce interference and improve the robustness of the network. On the contrary, when the input data is relatively clear and has strong regularity, the network may activate more neurons to enhance the classification ability. In a specific embodiment, as Figure 7As shown, during the process of adjusting the model structure, the optical scene information of the defective data is considered, and combined with the complexity of the defective data, the number of network layers is dynamically adjusted, the connection method is dynamically adjusted, and neurons are dynamically adjusted or deleted to finally output the classification result for the defective data. It should be noted that in a specific embodiment, the model structure can be visualized through an interactive interface to facilitate relevant personnel to intuitively understand the data classification process and make the network have a certain interpretability.

[0071] After adjusting the model structure through the above steps, the adjusted model is used to classify and predict the input defective data to obtain the corresponding classification and prediction results; further, the host computer can display the classification and prediction results of the defective data, and can display the high-definition pictures, defective attributes, scenes, defective category results predicted by the model, etc. of the defective data; so that the correctness of the classification results of the model can be reviewed manually. And after manual confirmation, the confirmed defective classification results can be returned to the model for retraining of the model. Specifically, through user feedback, the defective classification model can be continuously optimized to improve its accuracy and robustness; such as Figure 8 As shown, the data set for the classification and prediction results manually fed back can be added to the model for iterative optimization of the model. The core idea of the online learning mechanism is that after the model completes the classification task, it can further learn and optimize according to the correctness information fed back by the user. This mechanism enables the model to continuously adapt to new data during actual use and gradually improve its ability to classify defects. The adaptive defective classification system predicts the type of defect based on the input defective image data, and after outputting the predicted defective category label, it displays it to the user. After viewing the output results and relevant information of the model, the user can judge whether the output of the model is correct based on their professional knowledge and experience and promptly feedback it to the model for re-learning. The data set fed back by the user is input into the model, the weights and priorities are increased, and the classification model is iteratively optimized to gradually improve the accuracy and robustness of the model.

[0072] It can be seen that this application can determine the model structure information suitable for the to-be-classified picture data from the pre-trained initial classification model in combination with the complexity of the to-be-classified picture data, and then adjust the initial classification model to obtain the adjusted classification model corresponding to the model structure information, and use this adjusted classification model to perform classification prediction on the to-be-classified picture data; during the process, there is no need to re-train or pre-train multiple classification models, and the appropriate model structure can be selected in real time according to the differences of the to-be-classified picture data, and then the initial classification model can be dynamically adjusted according to the real-time model structure to obtain the adjusted classification model adapted to each classification prediction. In this way, different models are determined for classification prediction based on the complexity of the defect data, which can reduce the overfitting risk of the model on simple samples; it can be understood that when the model receives new data, it can adaptively select the most suitable network model for the defect classification task according to the samples, that is, according to the characteristics of the input data, including the distribution of attributes such as defect size, signal-to-noise ratio, and area, select the most suitable network model for the next classification task. The dynamic adjustment may include: dynamically adjusting the connection mode, dynamically adjusting the network layer, dynamically deleting neurons, and determining whether to output the predicted classification result. In this way, the computing resources and time can be optimized according to the different complexities during each data classification, the prediction efficiency of the model can be improved, and at the same time the classification accuracy can be improved. Moreover, this application can display the classification prediction result of the model and the relevant information of the input data (defect image, lighting condition, etc.) to the user; the user can judge whether the model output is correct according to professional knowledge or experience; then the data set fed back by the user can be input into the model to increase the weight and priority, and the classification model can be iteratively optimized to gradually improve the accuracy and robustness of the model.

[0073] Embodiment 4

[0074] Based on the content disclosed in the above embodiments, as Figure 9 shown, an embodiment of this application discloses a data classification device, including:

[0075] A complexity determination module 11, configured to determine the complexity corresponding to the to-be-classified picture data; the to-be-classified picture data is the picture data of the target object with defects in the wafer production process;

[0076] A model adjustment module 12, configured to adjust the pre-trained initial classification model based on the model structure information adapted to the complexity;

[0077] A classification prediction module 13, configured to perform classification prediction on the to-be-classified picture data by using the adjusted classification model.

[0078] It can be seen that the present application can determine the model structure information suitable for the image data to be classified from the pre-trained initial classification model in combination with the complexity of the image data to be classified, and then adjust the initial classification model to obtain the adjusted classification model corresponding to the model structure information, and use the adjusted classification model to classify and predict the image data to be classified; during the process, there is no need to re-train or pre-train multiple classification models, and the appropriate model structure can be selected in real time according to the differences of the image data to be classified, and then the initial classification model can be dynamically adjusted according to the real-time model structure to obtain the adjusted classification model adapted to each classification prediction; in this way, the adaptability of the classification model is improved, and the prediction efficiency can be improved.

[0079] In a specific embodiment, the complexity determination module 11 may include:

[0080] A complexity evaluation sub-module, configured to evaluate the complexity corresponding to the image data to be classified through at least one complexity evaluator in the initial classification model;

[0081] Wherein, different ones of the complexity evaluators are respectively preset at different skip connections in the initial classification model.

[0082] In another specific embodiment, the complexity evaluation sub-module may include:

[0083] A complexity evaluation unit, configured to extract features from the target defect image block through the complexity evaluator at the skip connection, and evaluate the complexity of the extracted features;

[0084] Wherein, the target defect image block is the defect image block corresponding to the image data to be classified received at the skip connection.

[0085] In a specific embodiment, the complexity evaluation unit is specifically configured to:

[0086] Process the extracted features based on a multi-layer perceptron;

[0087] Use a preset activation function to map the output result of the multi-layer perceptron to a preset complexity value range to obtain the complexity corresponding to the extracted features.

[0088] In another specific embodiment, the complexity evaluation unit is specifically configured to:

[0089] Determine the corresponding target features from the extracted features based on a preset complexity evaluation index;

[0090] Evaluate the complexity of the target features;

[0091] Among them, the extracted features include any one or several of the defect size information, energy focusing condition, area, aspect ratio, noise information, and resolution corresponding to the target defect image block.

[0092] In a specific embodiment, the model adjustment module 12 may include:

[0093] A scene information acquisition unit, configured to acquire the scene information of the first picture acquisition scene; the first picture acquisition scene is the picture acquisition scene corresponding to the picture data to be classified;

[0094] A model adjustment unit, configured to adjust a pre-trained initial classification model based on the model structure information adapted to the scene information and the complexity.

[0095] In another specific embodiment, the device may further include:

[0096] A data set acquisition module, configured to acquire a training data set collected in a second picture acquisition scene;

[0097] A model training module, configured to perform model training based on the training data set to obtain the initial classification model;

[0098] Among them, the second picture acquisition scene and the first picture acquisition scene are the same scene or different scenes.

[0099] In a specific embodiment, the model training module specifically includes:

[0100] A model training unit, configured to control model convergence based on a cross-entropy loss function and dynamically adjust the learning rate of each model parameter based on an adaptive moment estimation algorithm.

[0101] In a specific embodiment, the device may further include:

[0102] A data transmission module, configured to transmit target data content including the current classification prediction result to a preset interaction interface;

[0103] A feedback data acquisition module, configured to acquire user feedback data for the target data content on the preset interaction interface;

[0104] A model re-training module, configured to perform model re-training using a new training data set generated based on the user feedback data.

[0105] In another specific embodiment, the model re-training module may include:

[0106] A model retraining unit is used to retrain the initial classification model or the adjusted classification model using the new training data set.

[0107] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0108] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0109] Figure 10 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the data classification method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0110] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0111] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0112] Among them, the operating system 221 is used to manage and control each hardware device and computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the data classification method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0113] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the data classification method disclosed above is implemented. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0114] Furthermore, the present application also discloses a computer program product, including a computer program / instructions, which when executed by a processor, implement the data classification method as disclosed above.

[0115] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference can be made to the description of the method part for related parts.

[0116] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0117] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0118] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data classification method, characterized in that, Including: Determine the complexity corresponding to the image data to be classified; The image data to be classified is the image data of the target object with defects in the wafer production process; Based on the model structure information adapted to the complexity, adjust the pre-trained initial classification model; Use the adjusted classification model to perform classification prediction on the image data to be classified.

2. The data classification method according to claim 1, wherein The determination of the complexity corresponding to the image data to be classified includes: Evaluate the complexity corresponding to the image data to be classified through at least one complexity evaluator in the initial classification model; Among them, different complexity evaluators are respectively preset at different skip connections in the initial classification model.

3. The data classification method according to claim 2, characterized in that, The evaluation of the complexity corresponding to the image data to be classified includes: Through the complexity evaluator at the skip connection, extract features from the target defect image block and evaluate the complexity of the extracted features; Among them, the target defect image block is the defect image block corresponding to the image data to be classified received at the skip connection.

4. The data classification method according to claim 3, wherein The evaluation of the complexity of the extracted features includes: Process the extracted features based on a multi-layer perceptron; Use a preset activation function to map the output result of the multi-layer perceptron to a preset complexity value range to obtain the complexity corresponding to the extracted features.

5. The data classification method according to claim 3, characterized in that The evaluation of the complexity of the extracted features includes: Based on a preset complexity evaluation index, determine the corresponding target features from the extracted features; Evaluate the complexity of the target features; Among them, the extracted features include any one or several of the defect size information, energy focusing situation, area, aspect ratio, noise information, and resolution corresponding to the target defect image block.

6. The data classification method according to any one of claims 1 to 5, characterized in that The model structure information includes any one or several of the model output position information, model layer information, and model neuron information.

7. The data classification method according to any one of claims 1 to 6, characterized in that, The adjustment of the pre-trained initial classification model based on the model structure information adapted to the complexity includes: Obtain the scene information of the first image acquisition scene; the first image acquisition scene is the image acquisition scene corresponding to the image data to be classified; Based on the scene information and the model structure information adapted to the complexity, adjust the pre-trained initial classification model.

8. The data classification method according to claim 7, wherein, It also includes: Obtain the training data set collected in the second image acquisition scene; Perform model training based on the training data set to obtain the initial classification model; Among them, the second image acquisition scene and the first image acquisition scene are the same scene or different scenes.

9. The data classification method according to any one of claims 1 to 8, characterized in that During the training process of the initial classification model, it includes: Control the model convergence based on the cross-entropy loss function and dynamically adjust the learning rate of each model parameter based on the adaptive moment estimation algorithm.

10. The data classification method according to any one of claims 1 to 9, characterized in that After using the adjusted classification model to perform classification prediction on the image data to be classified, it also includes: Transmit the target data content including the current classification prediction result to a preset interaction interface; Obtain the user feedback data for the target data content on the preset interaction interface; Perform model retraining using the new training data set generated based on the user feedback data.

11. The data classification method according to claim 10, characterized in that The re-training of the model using the new training dataset generated based on the user feedback data includes: Using the new training dataset to re-train the initial classification model or the adjusted classification model.

12. A data classification device, characterized in that, Including: A complexity determination module for determining the complexity corresponding to the picture data to be classified; The picture data to be classified is the picture data of the target object with defects in the wafer production process; A model adjustment module for adjusting the pre-trained initial classification model based on the model structure information adapted to the complexity; A classification prediction module for classifying and predicting the picture data to be classified using the adjusted classification model.

13. An electronic device, characterized in that, Including: A memory for storing a computer program; A processor for executing the computer program to implement the data classification method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, For storing a computer program, the computer program, when executed by a processor, implements the data classification method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the data classification method according to any one of claims 1 to 11.

Images