Multi-expert system construction and training method and device for connectivity defects

Abstract

The application discloses a multi-expert system construction and training method and device for connectivity defects, which comprises the following steps: stratifying production line image data according to regions, process sections and defect types, and obtaining training samples through artificial review sample labeling; defining multiple experts according to data stratification and establishing corresponding semantic prototype sets to construct a defect detection model comprising a backbone network, a gating network and multiple experts; jointly training the model using the training samples, wherein the training target of the model at least includes a main task loss and a load balancing regularization term; positioning high confusion pairs for the trained model, constructing a difficult example dataset based on the high confusion pairs, and retraining related experts; adding new experts when a new process or drift alarm is detected, freezing the backbone network, and deploying the new experts in gray scale after calibrating the parameters of the new experts and the gating network. The scheme can solve the problems of unstable expert division and training, expert collapse and difficult examples that cannot be closed loop.

Description

Technical Field

[0001] This application relates to the field of industrial defect detection, and more specifically, to a method and apparatus for constructing and training a multi-expert system for connectivity defect detection. Background Technology

[0002] As the carrier of integrated circuits, wafers may exhibit various defects on their surfaces, such as cracks, impurities, and other abnormal features. These defects can adversely affect the electrical performance, reliability, and lifespan of the chip. Therefore, wafer defect identification is crucial in semiconductor manufacturing. However, with continuous optimization of manufacturing processes, the types of mixed defects generated are becoming increasingly complex and susceptible to noise interference, making it increasingly difficult to identify these defects using conventional methods.

[0003] Among them, connectivity defects exhibit strong morphological diversity and domain drift under different materials, linewidth levels, exposure, and noise conditions. Using a single model often results in instability in long-tailed scenarios; while using a multi-expert model can decompose according to the defect subspace, problems such as expert collapse (a few experts are over-selected while others hardly learn), unbalanced load, and difficulty in incrementally adding new experts can easily occur during training. Summary of the Invention

[0004] The main purpose of this application is to provide a method and apparatus for constructing and training a multi-expert system for connectivity defect detection, in order to solve problems such as unstable expert division and training, expert collapse, unbalanced load, inability to close loops for difficult examples, and inability to quickly expand for new processes / forms, and to enable efficient classification and detection of different types of defects.

[0005] To achieve the above objectives, the first aspect of this application proposes a method for constructing and training a multi-expert system addressing connectivity defects, including: The production line image data is stratified by region, process section, and defect type and morphology subclass, and training samples are obtained by manually reviewing the sample annotations; Define multiple experts according to the data layering dimension and establish a corresponding semantic prototype set to construct a defect detection model that includes a backbone network, a gating network and multiple experts; The backbone network, gating network, and various experts are jointly trained using training samples to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term. The trained model is evaluated and diagnosed to identify highly confusing pairs. A difficult example dataset is constructed based on these pairs and retrained using relevant experts. When a new process or drift alarm is detected, a new expert is added, the backbone network is frozen, and the parameters of the new expert and the gated network are calibrated before the new expert is deployed in a grayscale manner.

[0006] In one embodiment of this disclosure, production line image data is layered by region, process section, and defect type and morphology subclass, and training samples are obtained by manually reviewing sample annotations, including: Production line image data is stratified by region, process section, defect type, and morphological subclass. Defect types include short circuit, open circuit, open circuit, and pseudo-defect. Morphological subclasses include micro-bridging, burr continuity, fine broken lines, and local gaps. Based on the sample size and data characteristics of each stratum, the sample proportion of each stratum is divided, and a regression set covering the long-tail distribution is constructed. Independent statistical indicators are established for each stratum. For fuzzy samples or samples with uncertain model predictions in the regression set, manual verification and labeling are performed to establish high-confidence labels. These high-confidence labels are used to calibrate the model's prediction results and learn the rejection boundary.

[0007] In one embodiment of this disclosure, a defect detection model comprising a backbone network, a gating network, and multiple experts is constructed by defining multiple experts according to data hierarchical dimensions and establishing corresponding semantic prototype sets, including: Based on different defect types in the production line data, multiple experts are defined, each responsible for handling a specific type of defect. Key features that experts should focus on are defined through a semantic prototype set. A defect detection model is constructed, comprising a backbone network, a gating network, and multiple experts. The experts share a backbone network, which is responsible for extracting high-level features from the input data. The gating network is responsible for outputting the selection probability of each expert based on the semantic vector output by the backbone network and contextual information. The multiple experts are responsible for receiving feature vectors from the backbone network and predicting specific types of defects.

[0008] In one embodiment of this disclosure, training samples are used to jointly train the backbone network, the gating network, and various experts to obtain a trained model. The training objectives of the model include at least the main task loss and load balancing regularization terms, including: The backbone network and each expert are pre-trained. After pre-training, a gating network is introduced for joint training. During training, the routing dispersion is gradually increased. The model parameters are adjusted based on classification loss or segmentation loss, load balancing loss, routing entropy loss and diversity loss to obtain the trained model. Load balancing loss is used to make the average selection probability of each expert close to a uniform distribution. Routing entropy loss is used to ensure that the selection probability of the router output does not degenerate. Diversity loss is used to keep the outputs of different experts distinguishable in the feature space.

[0009] In one embodiment of this disclosure, evaluating and diagnosing the trained model, identifying highly confusing pairs, constructing a difficult example dataset based on the highly confusing pairs, and retraining the model with relevant experts includes: Collect predicted and true labels on the model validation set or actual running data to generate a confusion matrix; locate high confusion pairs based on the confusion matrix, establish a difficult example sampling strategy for experts corresponding to the high confusion pairs to obtain difficult example samples; perform targeted data augmentation on the difficult example samples, and retrain the model using the data-augmented difficult example samples.

[0010] In one embodiment of this disclosure, high-confusion pairs are located based on a confusion matrix, and a hard example sampling strategy is established for the experts corresponding to the high-confusion pairs to obtain hard example samples, including: The model identifies samples with a prediction confidence level below a preset threshold from the confusion matrix as low-confidence samples; the images in the training set are scaled, and samples with inconsistent predictions at different scales are sampled first; the distance between each sample and the semantic prototype of its class is calculated, and samples with high similarity to the prototype of its class are selected first.

[0011] In one embodiment of this disclosure, when a new process or drift alarm is detected, an expert is added, the backbone network is frozen, and the parameters of the new expert and the gating network are calibrated before the new expert is deployed in a grayscale manner. This includes: When a new process or data drift is detected, the parameters of the backbone network are frozen, and the parameters of the gating network and the new experts are updated using the new process or drift data. The usage ratio of the new experts is gradually opened up, and the expert utilization rate, routing entropy and detection accuracy indicators are monitored in real time. If any one of them fails to meet the standard, the system is rolled back to the previous version.

[0012] The second aspect of this disclosure provides a device for constructing and training a multi-expert system to address connectivity deficiencies, comprising: The data stratification module is used to stratify production line image data by region, process section, and defect type and morphology subclass, and to obtain training samples by manually reviewing sample annotations; The expert definition module is used to define multiple experts according to the data layer dimension and establish a corresponding semantic prototype set to build a defect detection model that includes a backbone network, a gated network and multiple experts. The joint training module is used to jointly train the backbone network, gating network and various experts using training samples to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term. The difficult example closed-loop module is used to evaluate and diagnose the trained model, locate highly confusing pairs, construct a difficult example dataset based on the highly confusing pairs, and retrain the model with relevant experts. The incremental expansion module is used to add experts when a new process or drift alarm is detected, freeze the backbone network, and deploy the new experts in a grayscale manner after calibrating the parameters of the new experts and the gated network.

[0013] A third aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method provided in any of the first aspects.

[0014] A fourth aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method provided in any of the first aspects.

[0015] This disclosure presents a method and apparatus for constructing and training a multi-expert system for connectivity defects. Multiple experts are defined based on data hierarchical dimensions, and a corresponding semantic prototype set is established, allowing each expert to focus on a specific type of defect. By jointly training the backbone network, gating network, and individual experts, and incorporating load balancing regularization, diversity, and routing entropy terms into the loss function, the model's stability and robustness are improved while optimizing the main task, avoiding excessive reliance on certain experts. Hard example mining driven by a semantic confusion matrix enables continuously improving closed-loop optimization. Incremental expert expansion and small-step routing calibration enable rapid integration and gray-scale deployment of new processes / forms, allowing the model to cope with complex changes in the production process. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart illustrating the multi-expert system construction and training method for connectivity defects provided in this application; Figure 2 A structural block diagram of the multi-expert system construction and training device for connectivity defects provided in this application; Figure 3 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Connectivity defects in production line images are affected by different conditions (such as material, linewidth, and exposure), exhibiting strong morphological diversity and domain drift in different environments. This means that the same defect may have different visual features in different scenarios. This diversity and domain drift make the training and inference process of a single model difficult, especially when facing long-tailed scenarios. The model often performs well on a few samples but fails to generalize effectively in most edge cases or long-tailed scenarios. This is mainly because the model tends to focus on learning the common features of most samples during training, while ignoring edge scenarios that have a lower frequency in the training data. A core advantage of using a multi-expert model is that it can decompose different defect subspaces, allowing each expert to focus on a specific subspace (such as different types of defects, different morphological features, etc.). However, during training, some experts are over-selected while others are almost unselected, resulting in most experts not learning effectively. Due to the uneven distribution of the dataset, some experts process a large amount of data while others process only a small number of samples, leading to an unbalanced use of computing resources. Furthermore, in incremental training, adding new experts may make it difficult to balance the workload of existing experts, resulting in the inability of new experts to integrate effectively.

[0020] This disclosure constructs a multi-expert system to specialize in handling different types of defects. Through joint training, expert diversity regularization, and load balancing, it improves the overall performance of the system, enabling efficient classification and detection of various defect types (such as short circuits, open circuits, open circuits, and pseudo-defects). Furthermore, it provides continuous optimization mechanisms such as hard example mining, expert retraining, and new process detection to ensure the model can cope with complex changes and challenges in the production process.

[0021] Figure 1This is a flowchart illustrating the multi-expert system construction and training method for connectivity defects provided in this application. First, in step S102, the production line image data is layered according to region, process section, and defect type and morphology subclass, and training samples are obtained by manually reviewing the sample annotations.

[0022] To fully model the distribution characteristics of various defects and improve the model's ability to identify boundary samples and long-tail categories, the original image data is layered in multiple dimensions. In one embodiment of this disclosure, the production line image data is layered by region, process segment, defect type, and morphological subclass. Different image regions (such as the center, edge, pad area, and trace area) may exhibit different defect patterns and noise interference, so the production line image data can be layered by region. Images from different process segments (such as exposure, electroplating, and etching) have significant distribution shifts, so they can be layered by process segment. Layering can also be done by defect type and morphological subclass. Defect types include short circuits, open circuits, open circuits, and pseudo-defects, while morphological subclasses include micro-bridging, burr conduction, fine broken lines, and local gaps, ensuring sufficient sample coverage for both common defects (such as short circuits and open circuits) and rare defects (such as fine filaments, holes, and artifacts).

[0023] Real-world production line defects exhibit a long-tail distribution, meaning common defects have abundant data, while rare defects have sparse data. Without addressing this, the model may be biased towards the most frequent category. We can divide the sample size and data characteristics of each stratum to create a regression set covering the long-tail distribution. Furthermore, to evaluate the effectiveness of the stratification strategy and the model's performance at different strata, independent statistical metrics should be established for each stratum, which may include: Accuracy: The proportion of samples that are correctly classified.

[0024] Recall: The ability of a model to identify positive samples.

[0025] Precision: The accuracy of the model in predicting samples as positive.

[0026] F1-score: The harmonic mean of precision and recall, suitable for evaluating imbalanced datasets.

[0027] AUC-ROC: Evaluates the model's performance at different thresholds to determine its classification ability.

[0028] For fuzzy samples or samples with uncertain model predictions in the regression set, manual verification and labeling are performed to establish high-confidence labels. These high-confidence labels are used to calibrate the model's predictions and learn rejection boundaries. High-confidence labels represent the model's highly reliable judgment on specific samples and can be confirmed by domain experts. Using these high-confidence labeled samples for model calibration improves the model's reliability in real-world scenarios. By analyzing the model's performance on these samples, the model's decision boundary, i.e., the rejection boundary, can be adjusted to ensure the model makes reasonable judgments when faced with uncertain samples. This process can be carried out through cross-validation or using a specific validation set to continuously optimize model performance and avoid misclassification of low-confidence samples.

[0029] Subsequently, in step S104, multiple experts are defined according to the data layering dimension and a corresponding semantic prototype set is established to construct a defect detection model that includes a backbone network, a gating network and multiple experts.

[0030] Based on different defect types (short circuit, open circuit, open circuit, pseudo-defect, etc.) in the production line data, multiple experts can be defined, each specializing in handling a specific type of defect. To effectively constrain the learning direction of each expert, a core set of semantic prototypes is defined for each expert. This set defines the key features that the expert should focus on. Representative features can be extracted from defect samples in the dataset and used as semantic prototypes. For each defect type, feature vectors from several typical samples can be selected as prototypes. The number of semantic prototypes for each expert is determined based on the complexity and diversity of the defect types. For example, 3 to 5 prototypes can be used to achieve better representation capabilities.

[0031] A defect detection model is constructed, comprising a backbone network, a gating network, and multiple experts. These experts share a single backbone network, such as ResNet, DenseNet, or Transformer, which is responsible for extracting high-level features from the input data. Sharing a backbone reduces model parameters, improves training efficiency, and ensures that different experts learn based on the same feature set. The gating network outputs the selection probabilities of each expert based on the semantic vectors output by the backbone network and contextual information. The gating network can use a lightweight network structure, such as fully connected layers or a small convolutional network. Multiple experts receive feature vectors from the backbone network and further process them to predict specific types of defects. These experts can be networks with different structures or parameters, designed to capture different types of features or patterns. For example, one expert might focus on image classification, while another focuses on object detection. Each expert's output can be tailored to the specific task requirements to provide more fine-grained predictions.

[0032] Next, in step S106, the backbone network, gating network and various experts are jointly trained using training samples to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term.

[0033] According to one embodiment of this disclosure, firstly, a shared backbone network and a single expert are pre-trained. The purpose of this step is to allow the backbone network to learn basic feature representations and enable the expert to make initial adjustments for a specific task. After pre-training, a gating network is introduced for joint training. At this point, the gating network dynamically selects an expert based on the input features, allowing the model to flexibly switch between different tasks. During training, the dispersion of the gating network is gradually increased to make its expert selection more explicit and avoid overly concentrated selection of a single expert. Model parameters are adjusted based on the main task loss, load balancing loss, routing entropy loss, and diversity loss. The main task loss depends on the task requirements; for classification tasks (such as defect classification), a classification loss (such as cross-entropy loss) is used; for similarity learning tasks (such as defect similarity matching), a similarity loss (such as contrast loss) is used.

[0034] To prevent some experts from having excessively high selection probabilities while others are overlooked, a load balancing term is added to the loss function to make the average selection probability of each expert close to... Here, N represents the number of experts, ensuring all experts are adequately trained. To prevent gating output degradation, a routing entropy term is added. This loss function encourages the router to output higher entropy values, ensuring a degree of randomness in the selection of multiple experts and preventing some experts from being completely ignored. To ensure that different experts remain distinguishable in the feature space, a diversity term is introduced. This term encourages experts to learn complementary information rather than repetitive features by evaluating the differences in their outputs. This can be achieved by calculating the distance between the outputs of different experts, prompting each expert to play a unique role in a specific feature space. During training, it is also necessary to ensure the consistency of the outputs of all experts, especially when different parts of the task are co-learned, ensuring that the results of each expert's output match each other within the global framework of the task.

[0035] According to one embodiment of this disclosure, a minimum expert utilization threshold can be set to ensure that each expert is selected at least a certain number of times, preventing some experts from being completely ineffective during training. By limiting the entropy value of the routes, overly random or uncertain route selection can be reduced, making the routes more explicit and stable. Gated temperature annealing can balance the randomness and determinism of expert selection. In the early stages of training, higher temperatures are used for more random expert selection, while gradually lowering the temperature as training progresses allows for the selection of more deterministic experts. This helps the model explore more possible knowledge in the early stages and focus more on deep learning in the later stages.

[0036] Through the above steps, the joint training of the expert system and the gating network can flexibly utilize multiple experts for efficient task processing while sharing the same backbone network. Simultaneously, by introducing regularization terms such as load balancing, routing entropy, and diversity, the stability and generalization ability of the model are ensured, and the phenomenon of expert collapse is avoided, thereby improving the overall model performance and robustness.

[0037] In step S108, the trained model is evaluated and diagnosed to identify highly confusing pairs. A difficult example dataset is constructed based on the highly confusing pairs and the model is retrained with relevant experts.

[0038] According to one embodiment of this disclosure, predicted labels and true labels can be collected on a model validation set or actual running data to generate a confusion matrix. The confusion matrix can display the comparison between the true category and the predicted category, and is presented by calculating true positives, false positives, true negatives, and false negatives.

[0039] Highly confusing pairs are identified based on the confusion matrix. A hard example sampling strategy is then established for these pairs by experts to obtain hard example samples. In the confusion matrix, highly confusing pairs typically exhibit a high error ratio between the predicted and actual labels for certain categories. For example, in defect detection tasks, the model might confuse "short circuit" with "contamination pseudo-defect." The confusion matrix helps identify these highly confusing category pairs by calculating the misclassification frequency and cross-error rate. The hard example sampling strategy includes: identifying samples from the confusion matrix whose model prediction confidence is below a preset threshold as low-confidence samples. These are typically samples with model output probabilities close to 0.5. These samples are characterized by difficulty in clear model classification and are prone to errors. Images in the training set are scaled, prioritizing samples with inconsistent predictions at different scales. The distance between each sample and the semantic prototype of its category is calculated, prioritizing samples with high similarity to the prototype of its category. These samples may be located at category boundaries or in ambiguous regions between categories.

[0040] Targeted data augmentation can be performed on difficult examples, and the model can be retrained using these augmented examples. Augmentation strategies may include: Geometric transformations: such as rotation, translation, and clipping, these methods help simulate defects at different angles and positions, enhancing the model's tolerance to positional changes.

[0041] Color transformation: Adjust parameters such as brightness, contrast, and saturation of the image to simulate the appearance of defects under different lighting conditions. - Noise addition: Add noise (such as Gaussian noise, salt and pepper noise, etc.) to the image to increase the model's adaptability to different noise environments.

[0042] Pseudo-defect generation: For certain defects that are difficult to capture (such as pollution pseudo-defects), more similar samples are generated through generative models or transformation algorithms to expand the diversity of the dataset.

[0043] Based on the results of hard example mining, hard example samples are regenerated and the model is retrained. This incorporates hard example samples into the training set, improving the model's ability to handle boundary samples and difficult-to-identify samples. During training, higher weights can be assigned to hard example samples to ensure that the model's performance on these samples is strengthened. Through multiple rounds of closed-loop retraining, the model gradually reduces misclassification of hard example samples and continuously improves its ability to identify long-tail and fuzzy categories.

[0044] Finally, in step S110, when a new process or drift alarm is detected, a new expert is added, the backbone network is frozen, and the parameters of the new expert and the gated network are calibrated before the new expert is deployed in a grayscale manner.

[0045] The model may face performance degradation risks in the following scenarios: introduction of new processes, materials, or process adjustments on the production line; data distribution drift (such as changes in input distribution, defect patterns, or image style); and model instability warnings triggered by anomaly monitoring or performance evaluation systems. When a new process or data drift is detected, a new expert is introduced, the parameters of the backbone network are frozen, and the parameters of the gating network and the new expert are updated using the new process or drifted data. For example, the top-k selection distribution of the gating network can be adjusted to make it more likely to select the new expert under specific feature inputs; reweighting strategies or sample augmentation can be used to increase the activation probability of the new expert during training with new data; at the same time, the distribution of existing experts should not change drastically to avoid disrupting the stability of the original system.

[0046] The usage rate of new experts is gradually increased, and the utilization rate, routing entropy, and detection accuracy are monitored in real time. If any of these indicators fail to meet the standards, the system is rolled back to the previous version. For example, the deployment of new experts is divided into different traffic segments (such as 5%, 20%, 50%, and 100%), and the performance of the new experts is evaluated in real time for each gray-scale segment.

[0047] After deployment, continuously monitor the following three types of metrics: Expert utilization rate checks whether new experts have been successfully activated to prevent them from being completely ignored; route entropy assesses whether the route distribution is too concentrated and whether there is a risk of degradation; accuracy metrics evaluate the actual predictive performance of the overall model.

[0048] If any of the following situations are found during the gray-scale phase, you should immediately roll back to the previous stable version and synchronously record the configuration and metric logs of this failed deployment for subsequent analysis: The utilization rate of new experts is far lower than expected (e.g., they are never selected); The router entropy drops sharply, and the model degenerates into a single-expert mode; Accuracy metrics are below the upper limit or significantly lower than the previous version; Anomaly alerts (such as an increase in the error rate of outlier identification, an increase in the false positive rate, etc.).

[0049] Therefore, by introducing experts and conducting small-step calibration training after detecting new processes or data drift, new data distributions can be adapted quickly.

[0050] This disclosure also provides an apparatus for constructing and training a multi-expert system oriented towards connectivity defects, used to implement the above-described method embodiments. (See reference...) Figure 2 , Figure 2 The structural block diagram of the multi-expert system construction and training device for connectivity defects provided in this application is shown. The device 200 includes: a data layering module 210, an expert definition module 220, a joint training module 230, a difficult example closure module 240, and an incremental expansion module 250.

[0051] Among them, the data layering module 210 is used to layer the production line image data according to region, process section and defect type and morphology subclass, and obtain training samples by manually reviewing the sample annotations; The expert definition module 220 is used to define multiple experts according to the data layer dimension and establish a corresponding semantic prototype set to build a defect detection model that includes a backbone network, a gated network and multiple experts. The joint training module 230 is used to jointly train the backbone network, the gated network and various experts using training samples to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term. The difficult example closed-loop module 240 is used to evaluate and diagnose the trained model, locate highly confusing pairs, construct a difficult example dataset based on the highly confusing pairs, and retrain the model with relevant experts. The incremental expansion module 250 is used to add experts when a new process or drift alarm is detected, freeze the backbone network, and deploy the new experts in a grayscale manner after calibrating the parameters of the new experts and the gated network.

[0052] In one or more embodiments of this disclosure, the data layering module 210 is specifically used for: layering production line image data by region, process section, defect type, and morphological subclass; the defect types include short circuit, open circuit, open circuit, and pseudo-defect; the morphological subclasses include micro-bridge, burr continuity, fine broken line, and local gap; dividing the sample proportion of each layer according to the sample size and data characteristics of each layer, constructing a regression set covering the long-tail distribution, and establishing independent statistical indicators for each layer; manually verifying and labeling fuzzy samples or samples with uncertain model predictions in the regression set, and establishing high-confidence labels so as to calibrate the model's prediction results and learn the rejection boundary through the high-confidence labels.

[0053] In one or more embodiments of this disclosure, the expert definition module 220 is specifically used to: define multiple experts based on different defect types in the production line data, each expert being used to handle a specific type of defect; define key features that the experts should focus on through a semantic prototype set; construct a defect detection model including a backbone network, a gating network, and multiple experts, with multiple experts sharing a backbone network, the backbone network being responsible for extracting high-level features from the input data; the gating network being responsible for outputting the selection probability of each expert based on the semantic vector output by the backbone network and contextual information; and multiple experts being responsible for receiving feature vectors from the backbone network and predicting specific types of defects.

[0054] In one or more embodiments of this disclosure, the joint training module 230 is specifically used for: pre-training the backbone network and each expert, and introducing a gating network for joint training after the pre-training is completed; gradually increasing the routing dispersion during the training process, adjusting the model parameters based on classification loss or segmentation loss, load balancing loss, routing entropy loss and diversity loss to obtain the trained model, wherein the load balancing loss is used to make the average selection probability of each expert close to a uniform distribution, the routing entropy loss is used to ensure that the selection probability output by the router does not degenerate, and the diversity loss is used to keep the outputs of different experts distinguishable in the feature space.

[0055] In one or more embodiments of this disclosure, the difficult example closed-loop module 240 is specifically used for: collecting predicted labels and true labels on the model validation set or actual running data to generate a confusion matrix; locating high confusion pairs based on the confusion matrix, establishing a difficult example sampling strategy for experts corresponding to the high confusion pairs, and obtaining difficult example samples; performing targeted data augmentation on the difficult example samples, and retraining the model using the data-augmented difficult example samples. Specifically, establishing the difficult example sampling strategy to obtain difficult example samples includes: identifying samples from the confusion matrix whose model prediction confidence is lower than a preset threshold as low-confidence samples; dividing the images in the training set into scales, prioritizing the sampling of samples with inconsistent predictions at different scales; calculating the distance between each sample and the semantic prototype of its category, and prioritizing samples with high similarity to the prototype of its category.

[0056] In one or more embodiments of this disclosure, the incremental expansion module 250 is specifically used to: introduce new experts when a new process or data drift is detected, freeze the parameters of the backbone network, update the parameters of the gating network and the new experts using the new process or drift data; gradually open up the usage ratio of the new experts, monitor the expert utilization rate, routing entropy and detection accuracy indicators in real time, and roll back to the previous version if any one of them fails to meet the standard.

[0057] The specific methods of execution of each module or unit in the above device embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0058] This disclosure also provides an electronic device, such as... Figure 3 As shown, the electronic device includes one or more processors 301 and a memory 302. Figure 3 Take processor 301 as an example.

[0059] The electronic device may also include an input device 303 and an output device 304.

[0060] The processor 301, memory 302, input device 303, and output device 304 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0061] Processor 301 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips. The general-purpose processor can be a microprocessor or any conventional processor.

[0062] The memory 302, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 301 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 302, thereby implementing the methods in the above-described method embodiments.

[0063] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the processing device operated by the server. Furthermore, memory 302 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories can be connected to a network connection device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] One or more modules are stored in memory 302 and, when executed by one or more processors 301, perform the methods shown in the above embodiments.

[0065] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes as described in the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory (FM), hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0066] In summary, the multi-expert system construction and training method and apparatus for connectivity defects provided in this disclosure define multiple experts based on data hierarchical dimensions and establish corresponding semantic prototype sets, allowing each expert to focus on specific types of defects. By jointly training the backbone network, gating network, and individual experts, and by incorporating load balancing regularization, diversity, and routing entropy terms into the loss function, the model ensures that while optimizing the main task, it avoids over-reliance on certain experts, thereby improving the model's stability and robustness. Through semantic confusion matrix-driven hard example mining, a closed-loop optimization mechanism for continuous improvement is achieved. Incremental expert expansion and small-step routing calibration enable rapid integration and gray-scale deployment of new processes / forms, allowing the model to cope with complex changes in the production process.

[0067] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0068] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0069] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing and training a multi-expert system to address connectivity defects, characterized in that, include: The production line image data is stratified by region, process section, and defect type and morphology subclass, and training samples are obtained by manually reviewing the sample annotations; Define multiple experts according to the data layering dimension and establish a corresponding semantic prototype set to construct a defect detection model that includes a backbone network, a gating network and multiple experts; The training samples are used to jointly train the backbone network, the gating network, and various experts to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term. The trained model is evaluated and diagnosed to identify highly confusing pairs. A difficult example dataset is constructed based on the highly confusing pairs and then retrained with relevant experts. as well as When a new process or drift alarm is detected, a new expert is added, the backbone network is frozen, and the parameters of the new expert and the gating network are calibrated before the new expert is deployed in a grayscale manner.

2. The multi-expert system building and training method for connectivity defect orientation according to claim 1, characterized in that, The production line image data is stratified by region, process section, and defect type and morphology subclass, and the training samples are obtained by manually reviewing the sample annotations. The production line image data is layered by region, process section, defect type and morphology subclass. The defect types include short circuit, open circuit, open circuit and pseudo-defect. The morphology subclasses include micro-bridge, burr conduction, fine broken line and local gap. Based on the sample size and data characteristics of each stratum, the sample proportion of each stratum is divided, a regression set covering the long-tail distribution is constructed, and independent statistical indicators are established for each stratum. For fuzzy samples or samples whose model predictions are uncertain in the regression set, manual verification and labeling are performed to establish high-confidence labels, so as to calibrate the model's prediction results and learn the rejection boundary through high-confidence labels.

3. The method for constructing and training a multi-expert system oriented towards connectivity defects according to claim 1, characterized in that, The step of defining multiple experts based on data hierarchical dimensions and establishing corresponding semantic prototype sets to construct a defect detection model including a backbone network, a gating network, and multiple experts includes: Based on the different defect types in the production line data, multiple experts are defined, each expert is used to handle a specific type of defect, and the key features that the experts should focus on are defined by a semantic prototype set. A defect detection model is constructed, comprising a backbone network, a gating network, and multiple experts. The multiple experts share a backbone network, which is responsible for extracting high-level features from the input data. The gating network is responsible for outputting the selection probability of each expert based on the semantic vector output by the backbone network and contextual information. The multiple experts are responsible for receiving feature vectors from the backbone network and predicting specific types of defects.

4. The method for constructing and training a multi-expert system oriented towards connectivity defects according to claim 1, characterized in that, The training samples are used to jointly train the backbone network, gating network, and various experts to obtain the trained model. The training objectives of the model include at least the main task loss and load balancing regularization terms, including: The backbone network and each expert are pre-trained, and after the pre-training is completed, the gating network is introduced for joint training. During training, the routing dispersion is gradually increased, and the model parameters are adjusted based on classification loss or segmentation loss, load balancing loss, routing entropy loss and diversity loss to obtain the trained model. The load balancing loss is used to make the average selection probability of each expert close to a uniform distribution, the routing entropy loss is used to ensure that the selection probability of the router output does not degenerate, and the diversity loss is used to keep the outputs of different experts distinguishable in the feature space.

5. The method for constructing and training a multi-expert system oriented towards connectivity defects according to claim 1, characterized in that, The step of evaluating and diagnosing the trained model, identifying highly confusing pairs, constructing a difficult example dataset based on the highly confusing pairs, and retraining the model with relevant experts includes: Collect predicted labels and true labels on the model validation set or actual running data to generate a confusion matrix; Based on the confusion matrix, high confusion pairs are located, and a difficult example sampling strategy is established for the experts corresponding to the high confusion pairs to obtain difficult example samples. Targeted data augmentation is performed on the difficult examples, and the model is retrained using the augmented difficult examples.

6. The method for constructing and training a multi-expert system oriented towards connectivity defects according to claim 5, characterized in that, The step of locating highly confused pairs based on the confusion matrix and establishing a difficult example sampling strategy for the experts corresponding to the highly confused pairs to obtain difficult example samples includes: Samples whose model prediction confidence is lower than a preset threshold are identified from the confusion matrix as low-confidence samples; The images in the training set are divided into scales, and samples with inconsistent predictions at different scales are sampled first. Calculate the distance between each sample and the semantic prototype of its category, and prioritize samples with high similarity to the prototype of its category.

7. The method for constructing and training a multi-expert system oriented towards connectivity defects according to claim 1, characterized in that, The process of adding a new expert upon detecting a new process or drift alarm, freezing the backbone network, calibrating the parameters of the new expert and the gating network, and then deploying the new expert in a grayscale manner includes: When a new process or data drift is detected, a new expert is introduced, the parameters of the backbone network are frozen, and the parameters of the gating network and the new expert are updated using the new process or drift data. The usage rate of the new experts will be gradually increased, and the utilization rate of experts, routing entropy and detection accuracy will be monitored in real time. If any one of them fails to meet the standard, the system will be rolled back to the previous version.

8. A multi-expert system construction and training device for connectivity defects, characterized in that, include: The data stratification module is used to stratify production line image data by region, process section, and defect type and morphology subclass, and to obtain training samples by manually reviewing sample annotations; The expert definition module is used to define multiple experts according to the data layer dimension and establish a corresponding semantic prototype set to build a defect detection model that includes a backbone network, a gated network and multiple experts. The joint training module is used to jointly train the backbone network, the gating network and various experts using the training samples to obtain the trained model. The training objectives of the model include at least the main task loss and the load balancing regularization term. The difficult example closed-loop module is used to evaluate and diagnose the trained model, locate highly confusing pairs, construct a difficult example dataset based on the highly confusing pairs, and retrain the model with relevant experts. The incremental expansion module is used to add experts when a new process or drift alarm is detected, freeze the backbone network, and deploy the new experts in a grayscale manner after calibrating the parameters of the new experts and the gated network.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the method for constructing and training a multi-expert system for connectivity defects as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method for constructing and training a multi-expert system for connectivity defects as described in any one of claims 1 to 7.

Images