Method and apparatus for reproducing rare events and predicting yield using AI agent-based intelligent sampling

Abstract

A simulation device for early prediction of the occurrence of rare events based on an artificial neural network model according to the present invention may include: a first computational processing unit that trains ensemble models by extracting samples from a training data set using a predetermined sampling technique, calculates the average prediction failure probability of the ensemble models for candidate samples, and estimates an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples; and a second computational processing unit that selects the next sample to be simulated based on the estimated average prediction failure probability and the uncertainty index.

Description

Technology Field

[0001] The present invention relates to an apparatus and method for early prediction of rare events, and more specifically, to a simulation apparatus and method for early prediction of the occurrence of rare events based on an artificial neural network model. Background Technology

[0002] As the complexity of semiconductor circuit design increases, the amount of computation and simulation time required during the simulation process are rising rapidly. In particular, for analog simulators, even when utilizing existing high-precision simulators based on SPICE or FastSPICE, there are limitations to computational speed, and there is a problem where significant time and resources are consumed during the overall design verification process.

[0003] When estimating the failure rate of designs with performance affected by variation, it is important to predict it efficiently, especially in high-yield semiconductor designs where failure is a rare event with low probability; however, it is true that conventional prediction methods have many limitations.

[0004] Traditional Monte Carlo simulations have the advantage of being simple to implement and applicable regardless of system complexity, as they estimate probabilities by counting the occurrences of events of interest through random sampling. However, Monte Carlo simulations require a very large number of samples for accurate estimation in the case of rare events, which leads to enormous computational costs. A critical drawback is that if the probability of a rare event is too low, a massive number of samples is required, causing computational costs to increase exponentially.

[0005] Traditional Monte Carlo (MC) simulations estimate the probability of failure by randomly sampling, but rare events (e.g., 10 -6 Level probability has the problem of inefficiency because failures rarely occur no matter how many samples are drawn.

[0006] Quasi-Monte Carlo (QMC) simulations use low-discrepancy sequences (e.g., Halton or Sobol sequences) instead of random numbers to fill the sample space more uniformly. This allows for faster convergence speeds compared to Monte Carlo simulations. Thus, there is an advantage in that it can provide faster results than Monte Carlo simulations for the same number of samples. However, a disadvantage of QMC is that generating and applying low-discrepancy sequences becomes complex in high-dimensional problems.

[0007] Next, Sigma Scaled Sampling (SSS) is a technique that concentrates samples in the 'tail' region where rare events occur, particularly in semiconductor designs with high process variability (sigma). This is typically achieved by manipulating the distribution of process variables to artificially increase the sample density in failure regions. While Sigma Scaled Sampling has the advantage of efficiently identifying areas with a high probability of failure, it has the disadvantage of high complexity, such as the need for a de-biasing process to correct results obtained from the manipulated distribution back to the original distribution.

[0008] Importance Sampling (IS) draws samples from an importance distribution q(x), which is more likely to have rare events, instead of the original distribution p(x). While theoretically an improvement, there are issues with the difficulty of setting an appropriate distribution, and if done incorrectly, it can actually worsen performance. Although importance sampling designs a specific distribution to ensure rare events occur frequently, there is a problem where selecting the wrong distribution can lead to unstable estimation.

[0009] As such, existing methods have limitations, such as wasting significant computational resources or having performance heavily dependent on the selection of distributions, and suffer from low efficiency in detecting rare events early. Accordingly, this invention proposes an intelligent active learning-based rare event prediction method utilizing artificial neural networks that overcomes the limitations of traditional sampling methodologies. Prior art literature

[0010] Korean Patent Publication No. 10-2022-0120778 The problem to be solved

[0011] The technical problem to be solved by the present invention is to provide a method for early prediction of the occurrence of rare events based on an artificial neural network model.

[0012] The technical problem to be solved by the present invention is to provide a simulation device that predicts the occurrence of rare events early based on an artificial neural network model.

[0013] Another technical objective of the present invention is to provide a computer-readable recording medium that records a program for executing on a computer a method for early prediction of the occurrence of rare events based on an artificial neural network model.

[0014] The technical problems to be solved by the present invention are not limited to the above technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0015] A method for early prediction of the occurrence of rare events based on an artificial neural network model according to the present invention, for achieving the above technical problem, may include the steps of: training ensemble models by extracting samples from a training data set using a bootstrap sampling technique; calculating the average prediction failure probability of the ensemble models for candidate samples; estimating an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples; and selecting the next sample to be simulated based on the estimated average prediction failure probability and the uncertainty index.

[0016] The above method further includes the step of estimating a feature space novelty index for the above candidate samples, wherein the further they are from the feature space in which the feature vectors of the learned samples are located, the more novel they are evaluated, and the step of selecting the next sample to be simulated may be selected by further considering the feature space novelty index.

[0017] The above method may further include the step of simulating the selected sample to be simulated through a simulator and outputting the result, and the step of training the output result on the ensemble models.

[0018] The step of selecting the next sample to be simulated may include: a step of calculating a normalized score for each of the candidate samples by normalizing the estimated average prediction failure probability and the uncertainty index, respectively; a step of calculating a final score for each of the candidate samples by summing weighted scores obtained by multiplying each normalized score by a pre-set corresponding weight; and a step of selecting the sample with the highest final score as the next sample to be simulated.

[0019] The step of selecting the next sample to be simulated may include: a step of calculating a normalized score for each of the candidate samples by normalizing the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index, respectively; a step of calculating a final score for each of the candidate samples by summing weighted scores obtained by multiplying each normalized score by a pre-set corresponding weight; and a step of selecting the sample with the highest final score as the next sample to be simulated.

[0020] The step of extracting and training the above samples may include the step of training by adjusting the weights higher and passing them to the loss function when the 'failure' class in the training dataset is less than or equal to a threshold of the 'success' class.

[0021] The above sample may be one or more input conditions that can be simulated to estimate performance or defect rate.

[0022] A simulation device for early prediction of the occurrence of rare events based on an artificial neural network model according to the present invention, for achieving other technical challenges mentioned above, may include: a first processing unit that trains each of the ensemble models by extracting samples from a training data set using a bootstrap sampling technique, calculates the average prediction failure probability of the ensemble models for candidate samples, and estimates an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples; and a second processing unit that selects the next sample to be simulated based on the estimated average prediction failure probability and the uncertainty index.

[0023] The simulation device may further include memory storing ensemble models and training data sets.

[0024] In the simulation device, the first computational processing unit estimates a feature space novelty index for the candidate samples, which evaluates novelty as the further they are from the feature space where the feature vectors of the learned samples are located, and the second computational processing unit can select the next sample to be simulated by further reflecting the feature space novelty index.

[0025] The simulation device may further include a simulator module that receives the selected next sample to be simulated, performs a simulation, and outputs a result.

[0026] The second processing unit above can control the first processing unit to train the output results on the ensemble models.

[0027] When the second processing unit selects the sample to be simulated next, for each of the candidate samples, the estimated average prediction failure probability and the uncertainty index are each normalized to calculate a normalized score, and a weighted score is calculated by multiplying each normalized score by a pre-set corresponding weight to sum up the results to calculate a final score for each of the candidate samples, and the sample with the highest final score can be selected as the sample to be simulated next.

[0028] When the above second processing unit selects the sample to be simulated next,

[0029] For each of the above candidate samples, the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index are each normalized to calculate a normalized score, and

[0030] A final score is calculated for each of the candidate samples by summing the weighted scores obtained by multiplying each of the above normalized scores by the corresponding pre-set weights, and the sample with the highest final score can be selected as the next sample to be simulated. Effects of the invention

[0031] According to the present invention, by synthesizing the average prediction failure probability, uncertainty, and novelty indicators to selectively simulate samples with high informational value, there is an advantage of being able to quickly secure new failure scenarios with a much smaller number of iterations compared to conventional random simulations.

[0032] In addition, the method for early prediction of rare events and the simulation device (100) for the same according to the present invention have the advantage of enabling early prediction or detection of rare events of risk factors in a wide range of industrial fields, such as safety systems and finance, as well as in the semiconductor field.

[0033] In addition, according to the present invention, while random search requires hundreds of thousands to millions of simulations, the present invention intensively executes only selected samples, thereby having the advantage of enabling accurate estimation while significantly reducing computational resources, time, and costs.

[0034] In addition, according to the present invention, there is an advantage in that generalization performance is secured by including samples with new patterns, thereby enabling the capture of various rare events without being limited to specific conditions.

[0035] In addition, according to the present invention, reliability based on uncertainty is secured, data imbalance is mitigated, and model generalization performance is improved, so there is an advantage of being able to accurately estimate the probability of rare events occurring early through active learning based on artificial neural network ensemble.

[0036] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0037] Figure 1 is a diagram illustrating the layer structure of an artificial neural network. Figure 2 is a diagram illustrating an example of a deep neural network. FIG. 3 is a block diagram illustrating the configuration of a simulation device (100) that predicts the occurrence of rare events early based on an artificial neural network model according to the present invention. FIG. 4 is a flowchart illustrating the process of a simulation device according to the present invention estimating the final failure probability through an ensemble model. Specific details for implementing the invention

[0038] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiment in which the present invention may be practiced. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will know that the present invention may be practiced without such specific details.

[0039] In some cases, to avoid obscuring the concept of the present invention, known structures and devices may be omitted or illustrated in the form of block diagrams focusing on the core functions of each structure and device. Additionally, throughout this specification, the same components are described using the same reference numerals.

[0040] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0041] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0042] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0043] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] Furthermore, the components of the embodiments described with reference to each drawing are not limited to the respective embodiments and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention; it is also obvious that multiple embodiments may be re-implemented as a single embodiment that integrates multiple embodiments, even if a separate description is omitted.

[0045] Additionally, terms such as “~part,” “~unit,” “~module,” and “~device” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0046] Before describing the present invention below, the artificial neural network model, which is an ensemble model used in the present invention, will be briefly explained first.

[0047] Deep learning is a type of artificial neural network (ANN) based on human neural network theory. It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) composed of a layer structure and having one or more hidden layers (hereinafter referred to as intermediate layers) between the input layer and the output layer. Simply put, deep learning can be described as an artificial neural network with deep layers.

[0048] Artificial neural networks (ANNs), a field of artificial intelligence, are mathematical models modeled by mimicking the structure of the biological (typically human) brain (neural networks). In other words, artificial neural networks are implemented by imitating the information processing and transmission processes of these biological neurons. As they are implemented similarly to how the human brain solves problems, neural networks possess excellent parallelism because each neuron operates independently. Furthermore, since information is distributed across numerous connections, problems in a few neurons do not significantly affect the entire network; consequently, they are resilient to a certain level of error and possess the ability to learn from a given environment.

[0049] Deep neural networks can be viewed as descendants of artificial neural networks. They are the latest version of artificial neural networks, having overcome existing limitations and achieved success in areas where numerous artificial intelligence technologies had previously failed. When examining the modeling of artificial neural networks that mimic biological neural networks, biological neurons are modeled as nodes in terms of processing units, and synapses are modeled as weights in terms of connections, as shown in Table 1 below.

[0050] biological neural networks artificial neural networks cell body node dendrites input Axon output synapse weight

[0051] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0052] Just as human biological neurons perform meaningful tasks by connecting multiple cells rather than just one, artificial neural networks connect individual neurons to one another through synapses, creating multiple interconnected layers where the connection strength between layers can be updated using weights. In this way, they are utilized in fields for learning and cognition through their multi-layered structure and connection strengths.

[0053] Each node is connected by weighted links, and the entire model learns by repeatedly adjusting these weights. Weights represent the importance of each node as a fundamental means for long-term memory. Simply put, an artificial neural network trains the entire model by initializing these weights and updating and adjusting them with the data set to be trained. Once training is complete, when a new input is received, it infers an appropriate output value. The learning principle of an artificial neural network can be viewed as the process by which intelligence is formed from the generalization of experience, and it operates in a bottom-up manner. In Figure 1, when there are two or more intermediate layers (i.e., 5 to 10), the layers are considered to be deep, and it is called a Deep Neural Network; the learning and inference model achieved through such a Deep Neural Network can be referred to as Deep Learning.

[0054] Artificial neural networks can perform a certain role even with only one intermediate layer (commonly referred to as a 'hidden layer') in addition to inputs and outputs, but as the complexity of the problem increases, the number of nodes or layers must be increased. Among these, adopting a multi-layered model by increasing the number of layers is effective, but its scope of application is limited due to the limitations that efficient learning is impossible and the amount of computation required to train the network is large.

[0055] However, as the existing limitations mentioned above have been overcome, artificial neural networks have become capable of adopting deep structures. This has enabled the construction of complex and highly expressive models, leading to the 발표 of groundbreaking results in various fields such as speech recognition, face recognition, object recognition, and character recognition.

[0056] Figure 2 is a diagram illustrating an example of a deep neural network.

[0057] A Deep Neural Network (DNN) is an Artificial Neural Network (ANN) composed of multiple hidden layers between an input layer and an output layer. It is a set of machine learning models or algorithms referring to a Deep Neural Network (DNN) that has one or more hidden layers between an input layer and an output layer. Connections in a neural network are formed from the input layer to the hidden layer, and from the hidden layer to the output layer.

[0058] Deep neural networks, like general artificial neural networks, can model complex non-linear relationships. For example, in a deep neural network structure for an object identification model, each object can be represented as a hierarchical composition of the basic elements of an image. In this case, additional layers can combine features from progressively gathered lower layers. This characteristic of deep neural networks enables the modeling of complex data with fewer units (nodes) compared to similarly performed artificial neural networks.

[0059] Previous deep neural networks were typically designed as feedforward networks, but recent research has successfully applied deep learning structures to Recurrent Neural Networks (RNNs). Examples include the application of deep neural network structures in the field of language modeling. In the case of Convolutional Neural Networks (CNNs), not only have they been successfully applied in the field of computer vision, but their successful applications are also well-documented. More recently, CNNs have been applied to acoustic modeling for Automatic Speech Recognition (ASR) and are considered to have been more successful than existing models. Deep neural networks can be trained using the standard backpropagation algorithm. In this process, weights can be updated through stochastic gradient descent.

[0060] The simulation device and method for early prediction of rare events based on an artificial neural network model proposed in this invention are not limited to specific industries such as electrical circuits or semiconductor design, but are applicable to all fields. The early prediction technology for rare events proposed in this invention can be applied to all industrial sectors, including manufacturing and quality control, disaster and safety simulation, new drug development and biotechnology, and financial risk analysis. It is particularly useful in fields dealing with "low probability, high risk" problems.

[0061] The term "sample" as used in this invention does not simply refer to a single data point, but rather to a specific instance of input conditions (e.g., design parameters, process parameters, environmental parameters, etc.) capable of performing a simulation. For example, in the case of semiconductor circuit design, it may be a combination of process parameters (e.g., threshold voltage, channel length, transistor width / length (W / L), supply voltage, ambient temperature at which the circuit operates, internal operating temperature of the chip, etc.). In this way, a single 'condition vector' to be input for performing a simulation can serve as a sample. Therefore, in this invention, a sample is not merely an 'arbitrary point in a data set,' but a data point that possesses significant informational value for model training and inference. Accordingly, in this invention, a sample refers to a specific combination of simulable input conditions (design variables, process variables, environmental conditions, etc.) for estimating the performance or failure rate of a target system / device; it signifies a data point that is input into a prediction model and serves as a unit for estimating failure probability, evaluating uncertainty, and calculating feature space novelty.

[0062] As described above, an artificial neural network (ANN) is a general term for a machine learning model that hierarchically organizes a number of computational nodes (neurons) and connections (weights) between them to learn patterns from input data and generate output results. The 'ensemble model' used in the present invention is an artificial neural network model and can take various forms, such as a Multilayer Perceptron (MLP) ensemble or a Transformer ensemble.

[0063] The Multilayer Perceptron (MLP) structure is a typical feed-forward neural network structure consisting of an input layer, hidden layers, and an output layer. The computation method involves calculating weights × inputs + biases in each layer and passing the result through an activation function (such as ReLU or Sigmoid). Each MLP model consists of a feature extractor and a classifier, and the ReLU activation function and Dropout prevent overfitting and improve the model's generalization performance. The feature extractor converts input features into more abstract representations, which can be utilized for subsequent novelty calculations. The classifier predicts failure probabilities (binary classification) based on the extracted features. Since BCEWithLogitsLoss is used, the sigmoid function is applied to the final output to convert it into a probability.

[0064] The Transformer is an artificial neural network structure that dynamically learns the relationships between elements within a sequence by combining a self-attention mechanism and a feedforward layer (MLP block) in the hidden layer.

[0065] The "ensemble model" used in the present invention may refer to a model that includes a plurality of artificial neural network models having the same structure, wherein each artificial neural network model is trained based on different subsets extracted from the same training data set through a bootstrap sampling technique. If all ensemble models only look at the same data, the ensemble predictions become nearly identical, and the "uncertainty indicator (variance between models)" disappears. When uncertainty disappears, the search area and the application area become unbalanced as a strategy for selecting the next sample. In situations where failure data is rare, by reinforcing the minority class (rare events), some ensemble models see a relatively large number of failure samples through bootstrapping, which allows the entire ensemble model to more stably reflect the influence of the failure class.

[0066] As such, the term "ensemble model" in this specification refers not merely to a collection of multiple artificial neural networks, but to a collection that simultaneously performs reliable prediction and uncertainty quantification functions based on model diversity.

[0067] The core of the rare event problem is that "failure" occurs probabilistically very rarely in high-reliability systems such as semiconductors, aviation, and energy. If simulations are performed using random sampling, as in the traditional Monte Carlo method, a massive number of simulations are required to accurately estimate the probability of failure because failures rarely occur. However, traditional approaches are inefficient in real industrial environments because simulation costs (time and computational resources) are very high.

[0068] In the present invention, the failure probability estimation is a model-based estimation that predicts the probability of a candidate sample failing using an artificial neural network ensemble model. Simulating samples with a high probability of failure allows for faster acquisition of actual failure cases, and as a result, the failure probability estimation is used as a criterion for determining which samples to simulate. By focusing on learning regions with a high probability of failure, efficiency is achieved in capturing the distribution of rare events with much fewer simulations than in random search.

[0069] As such, failure probability estimation is not merely about extracting probability values, but is a strategic tool for detecting failure scenarios early. By selecting samples based on average predicted failure probabilities and uncertainties, new failure modes can be rapidly identified, enabling the early prediction of rare events.

[0070] FIG. 3 is a block diagram illustrating the configuration of a simulation device (100) that predicts the occurrence of rare events early based on an artificial neural network model according to the present invention.

[0071] Referring to FIG. 3, the simulation device (100) may include a memory (110), a first operation processing unit (120), a second operation processing unit (130), and a simulation module (140). The simulation module (140) may include an input unit (145) and an operation unit (147).

[0072] As used in this specification, the term "first operation processing unit" (120) means including a processor (e.g., Graphic Processing Unit, GPU) capable of performing matrix operations in parallel and its dedicated memory (e.g., VRAM), and as used in this specification, the term "second operation processing unit" (130) may mean including a processor (e.g., Central Processing Unit, CPU) capable of performing general-purpose operations and system memory (RAM). The memory (110) may store information for various operations, such as ensemble models and training data sets according to the present invention, and information for various controls before learning or prediction (inference).

[0073] The first processing unit (110) can train ensemble models by extracting samples from each training data set using a bootstrap sampling technique. The first processing unit (110) performs ensemble training. For example, the train_ensemble function can call multiple train_single_models to control multiple MLP models to be trained with different bootstrap sample data. Thus, all models constituting the ensemble use BCEWithLogitsLoss((Binary Cross Entropy with Logits Loss)) in the same way, and class imbalance processing (weight application) is also applied to each. In this way, the entire ensemble model can compensate for data imbalance while ensuring diversity in individual ensemble models.

[0074] For example, there may be cases in a training dataset where "failure (positive)" samples are very few and "success (negative)" samples are very numerous. In such imbalanced situations, since the artificial neural network model achieves low loss by accurately predicting only the 'success' class—which accounts for the majority of the data—a problem arises where the minority 'failure' class is ignored, making early prediction of rare events difficult.

[0075] Accordingly, the first processing unit (120) dynamically assigns weights to solve the above problem. In the present invention, as an example, the weight of each class can be adjusted by adding a weight argument to the BCEWithLogitsLoss function. For example, the imbalance problem can be solved by calculating the class ratio of the current dataset in the _train_single_model function and assigning a higher weight to the minority class (failure class). For example, if there are 100 samples of the failure class and 10,000 samples of the success class, the first processing unit (120) gives a weight of 100 times to the samples of the failure class, and the ensemble model learns more without ignoring the failure cases.

[0076] In this way, the first processing unit (110) dynamically calculates weight arguments within the _train_single_model function and passes them to BCEWithLogitsLoss. This alleviates the imbalance problem by adjusting and assigning weights so that the ensemble model learns the minority class more importantly when the number of samples of the failure (positive) class in the training dataset is less than a predetermined threshold (i.e., significantly less).

[0077] In addition, the above weights can be set by the user in analysis_params, for example. When the focus is on exploration (when you want to find more new areas), the weights for the uncertainty index and feature space novelty index can be relatively higher than the average prediction failure probability weights, and when the focus is on utilization (when you want to intensively check areas that already have many failures), the weights for the average prediction failure probability can be relatively higher than the weights for the uncertainty index and feature space novelty index.

[0078] Here, BCEWithLogitsLoss is a software function of a deep learning framework such as PyTorch and can be executed in the first processing unit (120) or the second processing unit (130). _train_single_model is a software function responsible for training individual artificial neural network models and calls BCEWithLogitsLoss internally. That is, the mathematical operation of BCEWithLogitsLoss (sigmoid transformation + binary cross-entropy calculation) can be performed as a vector matrix operation in the first processing unit (120). The entire ensemble consists of multiple models trained individually, and all models use the same imbalanced loss function.

[0079] The first computational processing unit (120) estimates the average prediction failure probability of the ensemble models for candidate samples. The average prediction failure probability is the degree to which the ensemble models commonly predict a high probability of failure. Each individual ensemble model can output / predict the probability of 'failure' by passing the input sample through a feature extractor and classifier process. Subsequently, the first computational processing unit (120) can estimate the average prediction failure probability by calculating the average of the failure probabilities predicted by each ensemble model. Samples with a high average prediction failure probability are preferred, which contributes to searching for failure areas.

[0080] Here, the candidate samples are different from the samples trained through bootstrap sampling. The samples extracted and trained by bootstrap sampling are a labeled dataset in which "success / failure" labels have already been determined by running a simulation. The first processing unit (120) creates a subset from this labeled (or labeled) dataset using random sampling with replacement, and trains each individual ensemble model (e.g., MLP).

[0081] Candidate samples are unlabeled input condition candidates that have not yet been simulated. Candidate samples are used by ensemble models to perform predictions and calculate failure probability, uncertainty index, and novelty index. In other words, candidate samples are samples that are likely to be selected as the next simulation target. These candidate samples can be generated in various ways, such as random, QMC, and Latin hypercubes, in the design and process parameter spaces.

[0082] The first computational processing unit (120) can estimate the prediction deviation (e.g., variance, standard deviation, etc.) between ensemble models for candidate samples, and this prediction deviation between ensemble models can be referred to as the uncertainty index. Samples with a large deviation between the predicted probabilities of the ensemble models are preferred. The reason samples with a large prediction deviation between ensemble models are preferred is that this is important for 'exploring' areas that the ensemble models are not yet certain of, and through this, it is effective to discover unknown failure modes.

[0083] The first computational processing unit (120) can calculate a feature space novelty index for candidate samples based on the distance from the feature space of the learned samples. Input data (e.g., process parameters, design parameters, etc.) is converted into embeddings in the form of high-dimensional vectors as it passes through the feature extractor of the ensemble model, and the space where these vectors are gathered is the feature space. Therefore, the "feature space" can be described as a vector space that abstractly represents the input data, that is, the learned samples. In this way, the feature space is the space where the feature vectors of the learned samples are located.

[0084] Feature space novelty, for example, can be calculated using the _calculate_feature_novelty function and measures how much new information a new sample provides based on feature space similarity (e.g., Radial Basis Function (RBF) kernel) with samples selected so far. A higher score indicates that the sample has a feature distribution different from the data learned so far, thereby contributing to improving the generalization ability of the ensemble model. Thus, the further a candidate sample is from the feature space of the learned samples, the more novel it is evaluated to be. The RBF kernel is calculated based on the distance between two feature vectors (e.g., Euclidean distance), and similarity is evaluated as high if two sample vectors are close in the feature space and low if they are far apart. In this invention, a newly selected sample is judged to be a "highly novel sample" if its RBF kernel similarity with existing training data is low—that is, if it is far from the feature space. By doing so, data from areas previously uncovered can be secured, thereby significantly improving the generalization performance of the ensemble model.

[0085] As such, since failure prediction ability and exploration / exploration of new areas are key indicators for early prediction of rare events, the second computational processing unit (130) helps in predicting rare events by selecting the next sample to simulate by considering the average prediction failure probability of the ensemble models and the uncertainty index between the ensemble models.

[0086] In this way, when the second processing unit (130) selects the next sample to be simulated, for each of the candidate samples, it can normalize the estimated average prediction failure probability and uncertainty index and then calculate the normalized score. Since the average prediction failure probability is always in the range of 0 to 1, but the uncertainty index is an indicator of a different nature, it is necessary to convert these indicators to a common scale to balance the numerical ranges. Therefore, the second processing unit (130) can normalize the estimated average prediction failure probability and uncertainty index and calculate the normalized score for each.

[0087] The second computational processing unit (130) sums the weighted scores obtained by multiplying each normalized score (normalized score) by a pre-set corresponding weight. At this time, the corresponding weights for the estimated average prediction failure probability and uncertainty index can be set by the user and may have different values. The second computational processing unit (130) can calculate the final score by summing the weighted scores for each of the candidate samples. The second computational processing unit (130) can select the sample with the highest final score as the sample to be simulated next.

[0088] Meanwhile, although only two indicators, the average prediction failure probability and the uncertainty index, were considered above, if the feature space novelty index, which is a third indicator according to the present invention, is not reflected as an indicator to be considered when selecting the next sample to be simulated, the data diversity (generalization performance) may be somewhat reduced. Therefore, preferably, the second computation processing unit (130) can select the next sample to be simulated by further reflecting the feature space novelty index.

[0089] In this case, similarly, since the feature space novelty index is an indicator of a different nature from the average prediction failure probability and uncertainty index, a process of normalizing and scoring is required. When the second computational processing unit (130) selects the next sample to be simulated, it can calculate a normalized score for each of the candidate samples by normalizing the estimated average prediction failure probability, uncertainty index, and feature space novelty index, respectively. The second computational processing unit (130) can calculate a final score for each of the candidate samples by summing the weighted scores obtained by multiplying each normalized score by a pre-set corresponding weight.

[0090] In this case, the corresponding weights for the estimated average prediction failure probability, uncertainty index, and feature space novelty index can be set by the user and may have different values. For example, if exploration is emphasized, the weights for the uncertainty index and feature space novelty index are increased, while if utilization is emphasized, the weight for the average failure probability can be relatively higher.

[0091] These three indicators, including the aforementioned average prediction failure probability, uncertainty index, and feature space novelty index, are complementary to each other, so it is desirable to reflect all of them when selecting the next sample to be simulated; however, considering the lightweighting of the simulation device (100) and the specificity of rare events in the relevant industrial field, only two indicators may be reflected. In this case, the two indicators may be three combinations of selecting two indicators from the three indicators above.

[0092] The simulation module (140) may include an input unit (145) and a calculation unit (147). The input unit (145) may receive a sample selected as the next sample to be simulated from the second calculation processing unit (130). The calculation unit (147) may perform a simulation on the received sample and output a result (e.g., final failure probability).

[0093] The second computational processing unit (130) can transmit the results of the simulation module (140) to the first computational processing unit (120) so that the first computational processing unit (120) can control the results output from the simulation module (140) to be trained on each of the ensemble models.

[0094] The simulation device (100) according to the present invention can perform a preprocessing process to prepare a learning step and a prediction step (evaluation of samples) for ensemble models. This is a process that must be undergone before retraining the model after adding new data to the training data set, and before inputting candidate samples into the ensemble model for prediction. The preprocessing process can largely consist of data normalization and data movement minimization processes.

[0095] Data normalization (applying StandardScaler) adjusts the distribution of input feature values ​​to a mean of 0 and a variance of 1, ensuring stable model training. This is essential during the training phase, and candidate samples are subsequently transformed using the same scaler during the inference phase. As an example, input features (xs) can be normalized using StandardScaler. This scaler is fitted to a randomly selected sample size (scaler_fit_size) from the entire dataset, thereby ensuring robustness against bias in the initial training dataset.

[0096] Next, data movement minimization (GPU memory management) minimizes performance degradation due to data transmission by not bringing the results after computation from the first computation unit (120) in the predict_ensemble_probs and get_ensemble_features functions back to the second computation unit (130), but keeping them in the first computation unit (120) as much as possible. Data is kept in the first computation unit (120) without unnecessarily moving it between the first computation unit (120) and the second computation unit (130). This is an optimization process to prevent performance degradation during iterative computations during training and during large-scale inference of candidate samples during prediction.

[0097] FIG. 4 is a flowchart illustrating the process of a simulation device (100) according to the present invention estimating the final failure probability through an ensemble model.

[0098] Referring to FIG. 4, the simulation device (100) can perform an initial sampling process by randomly selecting a small number of failure and success samples and normalizing them by applying StandardScaler, etc. (S410). The simulation device (100) can perform a process of training ensemble models by extracting samples from each training data set using a bootstrap sampling technique (S420).

[0099] Subsequently, the simulation device (100) can estimate the probability of failure for candidate samples through learned ensemble models (S430). In addition, the simulation device (100) can estimate an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples (S430). Furthermore, the simulation device (100) can calculate a feature space novelty index for the candidate samples (S430).

[0100] The simulation device (100) can normalize each calculated indicator and score it, and then calculate a final score for the candidate samples by multiplying each indicator by a corresponding weight and summing them (S440). The simulation device (100) can select the next sample to simulate based on the final score (S440).

[0101] The simulation device (100) can estimate the final failure probability by simulating the next sample to be simulated (S450). Subsequently, the simulation device (100) can repeat step S450 in step S420. In step S450, the simulation device (100) can perform a convergence decision process to determine whether to stop sample selection and additional learning by comparing the final failure probability estimated immediately before with the final failure probability estimated before that, and determining that the estimated rare event probability or the uncertainty of the ensemble model is no longer improved (S460).

[0102] For example, if the difference between the final failure probability estimated immediately before step S450 and the final failure probability estimated prior to that is below a predetermined threshold, it is determined that convergence has occurred, and the simulation device (100) may terminate (S460). Alternatively, if the difference between the final failure probability estimated immediately before and the final failure probability estimated prior to that is above a predetermined threshold, the simulation device (100) may repeat step S450 in step S420.

[0103] According to the method for early prediction of rare events and the simulation device (100) for performing the same according to the present invention described above, by synthesizing the average prediction failure probability, uncertainty, and novelty indicators and selectively simulating samples with high informational value, there is an advantage of being able to quickly secure new failure scenarios with a much smaller number of attempts compared to conventional random simulations.

[0104] In addition, the method for early prediction of rare events and the simulation device (100) for the same according to the present invention have the advantage of enabling early prediction or detection of rare events of risk factors in a wide range of industrial fields, such as safety systems and finance, as well as in the semiconductor field.

[0105] In addition, according to the present invention, while random search requires hundreds of thousands to millions of simulations, the present invention intensively executes only selected samples, thereby having the advantage of enabling accurate estimation while significantly reducing computational resources, time, and costs.

[0106] In addition, according to the present invention, there is an advantage in that generalization performance is secured by including samples with new patterns, thereby enabling the capture of various rare events without being limited to specific conditions.

[0107] In addition, according to the present invention, reliability based on uncertainty is secured, data imbalance is mitigated, and model generalization performance is improved, so there is an advantage of being able to accurately estimate the probability of rare events occurring early through active learning based on artificial neural network ensemble.

[0108] The processing device in the present invention may perform an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing device may be described as being used as a single unit, but those skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0109] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computing devices and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0110] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software.

[0111] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiments, and vice versa.

[0112] The embodiments described above are combinations of the components and features of the present invention in a specific form. Each component or feature should be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features.

[0113] Additionally, it is possible to construct embodiments of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment. It is obvious that embodiments may be constructed by combining claims that are not explicitly related in the claims, or that new claims may be included by amendment after filing.

[0114] It is obvious to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential features of the invention. Accordingly, the foregoing detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A method for a simulation device to select samples for early prediction of the occurrence of rare events based on an artificial neural network model comprises: a step of training ensemble models by extracting samples from a training data set using a predetermined sampling technique; a step of calculating the average prediction failure probability of the ensemble models for candidate samples; a step of estimating an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples; a step of estimating a feature space novelty index for the candidate samples, wherein the further the candidate samples are from the feature space in which the feature vectors of the trained samples are located, the more novel the candidate samples are evaluated; a step of selecting a sample to be simulated next based on the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index; a step of simulating the selected sample; and a step of estimating the final failure probability by reflecting the simulation results. A method comprising the steps of: stopping sample selection and additional learning when the difference between the final failure probability and the previously estimated final failure probability is below a predetermined threshold, and training the ensemble models when the difference between the final failure probability and the previously estimated final failure probability exceeds a predetermined threshold; estimating the average predicted failure probability, the uncertainty index, and the feature space novelty index; selecting the next sample to be simulated; simulating the selected sample; and repeating the step of estimating the final failure probability. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 The method according to claim 1, wherein the step of selecting the next sample to be simulated comprises: a step of calculating a normalized score for each of the candidate samples by normalizing the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index, respectively; a step of calculating a final score for each of the candidate samples by summing weighted scores obtained by multiplying each normalized score by a pre-set corresponding weight; and a step of selecting the sample with the highest final score as the next sample to be simulated. Claim 6 In claim 1, the method wherein the predetermined sampling technique is a bootstrap sampling technique. Claim 7 A method according to claim 1, wherein the step of extracting and training the sample includes the step of training the ensemble models by adjusting the weight for the 'failure' class to be higher and passing it to a loss function when the 'failure' class in the training data set is less than or equal to a threshold of the 'success' class. Claim 8 A method according to claim 1, wherein the sample is one or more input conditions that can be simulated to estimate performance or defect rate. Claim 9 A simulation device for selecting samples to predict the occurrence of rare events early based on an artificial neural network model, comprising: a first computational processing unit that trains ensemble models by extracting samples from a training data set using a predetermined sampling technique, calculates the average prediction failure probability of the ensemble models for candidate samples, estimates an uncertainty index related to the prediction deviation between the ensemble models for the candidate samples, and estimates a feature space novelty index for the candidate samples, wherein the further the candidate samples are from the feature space in which the feature vectors of the trained samples are located, the more novel the candidate samples are evaluated; a second computational processing unit that selects the next sample to be simulated based on the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index; and a simulator module that simulates the selected sample and estimates the final failure probability by reflecting the simulation results. A simulation device in which, if the difference between the final failure probability and the previously estimated final failure probability is below a predetermined threshold, the second processing unit stops sample selection and additional learning, but if the difference between the final failure probability and the previously estimated final failure probability exceeds a predetermined threshold, the first processing unit repeatedly performs training the ensemble models, estimating the average predicted failure probability, the uncertainty index and the feature space novelty index, selecting the next sample to be simulated, simulating the selected sample, and estimating the final failure probability. Claim 10 A simulation device according to claim 9, wherein the first processing unit, when extracting and training the sample, adjusts the weight for the 'failure' class to be higher and passes it to the loss function when the 'failure' class in the training data set is less than or equal to a threshold, thereby training the ensemble models. Claim 11 delete Claim 12 delete Claim 13 A simulation device according to claim 9, wherein when the second processing unit selects the next sample to be simulated, for each of the candidate samples, the estimated average prediction failure probability, the uncertainty index, and the feature space novelty index are each normalized to calculate a normalized score, and each of the candidate samples is given a weighted score by multiplying each normalized score by a pre-set corresponding weight to calculate a final score for each of the candidate samples, and the sample with the highest final score is selected as the next sample to be simulated. Claim 14 In claim 9, the simulation device wherein the predetermined sampling technique is a bootstrap sampling technique. Claim 15 A computer-readable recording medium having a program for executing a method described in any one of paragraphs 1, 5, 6, 7 and 8 on a computer.

Images