Lightweight method for yolo chip defect detection model based on topology-aware multi-object structured pruning

Abstract

The application discloses a YOLO chip defect detection model lightweight method using topology-aware multi-object structured pruning. The method acquires a to-be-pruned YOLO detection model, original weights and a chip defect detection dataset, and establishes an original model baseline; the topology connection relationship between a backbone network, a neck network and a detection head is analyzed, structures with channel dependency or feature fusion relationship are divided into a plurality of topology pruning groups, and a group-level pruning vector is constructed with the channel retention proportion of each group; protection constraints are set for a shallow high-resolution feature path and a small target detection path, proxy coarse screening, quick eval fine screening and NSGA-II multi-object search are used to obtain Pareto candidate structures; finally, a lightweight YOLO chip defect detection model is obtained through structured channel pruning, weight inheritance and formal fine tuning. The application can reduce the model parameter quantity, calculation quantity and inference delay, and is suitable for resource-limited industrial vision detection scenes.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning model compression, lightweight object detection model, and industrial vision inspection technology. Specifically, it relates to a lightweight YOLO model method utilizing topology-aware multi-objective structured pruning, and the application of the lightweight YOLO model in the detection of small defects on chip surfaces. This method reduces model computational complexity and deployment / inference latency while maintaining the ability to detect small defects in chips through network topology analysis, group-level pruning coding, shallow feature protection, phased evaluation of candidate structures, and multi-objective search. Background Technology

[0002] As semiconductor manufacturing processes advance towards higher integration and refinement, chip surface defect detection has become a crucial step in ensuring product quality and production yield. Chips are prone to defects such as cracks, breakage, contamination, and missing parts during manufacturing, dicing, packaging, and transportation. These defects are typically characterized by small size, weak edge features, and complex shapes. Traditional manual inspection methods are inefficient and easily affected by personnel experience and environmental factors, making it difficult to meet the speed and stability requirements of industrial production.

[0003] In recent years, deep learning-based object detection methods have been widely applied in the field of industrial defect detection. Among them, the YOLO series models have achieved good results in chip defect detection tasks due to their advantages such as fast detection speed and convenient deployment. However, in order to improve detection accuracy, existing YOLO models usually adopt deep network structures and complex multi-scale feature fusion mechanisms, resulting in high model parameter count, computational load, and inference latency, making them difficult to deploy efficiently in resource-constrained scenarios such as industrial control computers, embedded devices, and edge terminals.

[0004] Network pruning is a crucial technique for achieving lightweight models. However, most existing pruning methods employ independent layer-by-layer pruning or manually set pruning rates, lacking a holistic consideration of the YOLO multi-scale topology. This can easily disrupt the channel coordination of feature fusion paths, leading to decreased model accuracy. Furthermore, chip defect detection is a typical small-object detection task, heavily reliant on shallow, high-resolution features. Existing methods lack protection mechanisms for critical feature paths, easily resulting in missed detections of minute defects. Moreover, using only parameter count or GFLOPs as optimization targets fails to accurately reflect the model's performance on real-world hardware, while comprehensive training and evaluation of numerous candidate structures incurs significant computational overhead.

[0005] Therefore, a lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning is needed to reduce model complexity and inference latency while ensuring detection accuracy, thereby improving the deployment efficiency of the model in real industrial scenarios. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a lightweight method for YOLO chip defect detection models utilizing topology-aware multi-objective structured pruning. This invention yields a lightweight YOLO chip defect detection model suitable for deployment; it reduces the number of model parameters, computational load, and inference latency, making it suitable for resource-constrained industrial vision inspection scenarios and improving efficiency.

[0007] The present invention achieves the above technical objectives through the following technical means:

[0008] A lightweight method for YOLO chip defect detection model utilizing topology-aware multi-objective structured pruning includes the following steps:

[0009] Step 1: Using the original YOLO detection model to be pruned, the original model weights, and the chip defect detection dataset as input, establish the original model baseline;

[0010] Step 2: Analyze the topological connection relationship between the backbone network, neck network and detection head, divide the structure with channel dependency relationship or feature fusion relationship into several topological pruning groups, and use the channel retention ratio of each topological pruning group as the group-level pruning coding variable.

[0011] Step 3: Based on the dependence of chip micro-defects on shallow high-resolution features, channel retention constraints are set for shallow protection groups and small target detection related paths, and channel coordination correction is performed on candidate pruning structures; then, candidate pruning structures are evaluated in stages through proxy coarse screening and quick eval fine screening.

[0012] Step 4: Use the NSGA-II multi-objective genetic algorithm to search for the final pruning structure that balances detection accuracy, theoretical computational cost, and actual inference latency.

[0013] Step 5: Perform structured channel pruning on the original YOLO detection model based on the final pruning structure to obtain a lightweight YOLO chip defect detection model suitable for deployment.

[0014] Furthermore, step 1 includes the following steps:

[0015] Step 1.1: Obtain the original YOLO detection model, the original model weights, and the chip defect detection dataset; the chip defect detection dataset includes chip defect images and defect type labels;

[0016] Step 1.2: Train, validate, and test the original YOLO detection model using the chip defect detection dataset to obtain the baseline performance metrics of the unpruned model. These baseline performance metrics include detection accuracy, number of parameters, theoretical computational cost, model size, and actual inference latency; expressed as:

[0017] ;

[0018] in, This indicates the detection accuracy of the original model. This represents the theoretical computational cost of the original model. This represents the number of parameters in the original model. This represents the average inference latency of the original model. This indicates the size of the original model.

[0019] Furthermore, step 2 includes the following steps:

[0020] Step 2.1: Read the network structure file of the YOLO model to be pruned, analyze the connection relationship between the backbone network, neck network and detector head in the model, and identify the pruning structures in the model such as convolutional layers, feature extraction modules, feature fusion modules and front feature layers of the detector head;

[0021] Step 2.2: Based on the feature map scale, upstream and downstream connectivity, multi-scale feature fusion path, and detection head input relationship, divide the model to be pruned into K topology pruning groups;

[0022] Step 2.3: Using the channel retention ratio of each topological pruning group as the encoding variable for the individual in the genetic algorithm, construct the group-level pruning vector, represented as: ; This represents the channel retention ratio of the k-th topology pruning group. , This represents the minimum channel retention rate for the k-th topology pruning group. This represents the highest channel retention rate in the k-th topology pruning group;

[0023] Step 2.4: Based on the channel retention ratio of each topology pruning group, calculate the number of channels that should be retained in each group after pruning; and align the calculated channel numbers. The channel alignment formula is expressed as:

[0024] ;

[0025] in, Indicates the first The number of channels in a topological pruning group before pruning. This indicates the number of channels after pruning. This represents the channel alignment function, used to adjust the number of pruned channels to an integer multiple that satisfies network connectivity and hardware computation requirements;

[0026] Step 2.5: Based on the candidate pruning vectors and the number of channels in each topology pruning group, perform structural mapping on the network structure corresponding to the original YOLO model to generate candidate pruning models, and check the structural legality and constructibility of the candidate pruning models.

[0027] Furthermore, step 3 includes the following steps:

[0028] Step 3.1: Based on the characteristics of chip defect targets, the shallow backbone feature path, the shallow small target feature path, and the topology pruning group related to high-resolution feature fusion are identified as the shallow protection group, and a minimum channel retention ratio is set for the shallow protection group; the shallow protection constraint is expressed as: ;in, This represents a set of shallow protection groups. Indicates the minimum retention ratio in the shallow layer. For the k-th topological pruning group, Represents topological pruning groups The corresponding channel retention ratio;

[0029] Step 3.2: Based on meeting the minimum shallow channel retention ratio, a shallow soft protection penalty term is set during the candidate structure screening process. When the channel retention ratio of the shallow protection group is lower than the preset soft protection target value, a penalty is applied to the candidate pruned structure, so that the genetic search process prioritizes the retention of shallow high-resolution feature paths that are important for the detection of minute defects. The shallow protection penalty term is expressed as follows:

[0030] ;

[0031] in, This indicates a shallow protection penalty. This represents a set of shallow protection groups. This represents the target value for shallow soft protection. Indicates the first The channel retention ratio of each topology pruning group; when Below In such cases, the candidate structure is penalized accordingly;

[0032] Step 3.3: Perform a proxy coarse screening on the candidate pruning structures generated by the genetic algorithm.

[0033] Step 3.4: Perform quick eval fine screening on the candidate pruned structures that have passed the proxy coarse screening;

[0034] Step 3.5: After the candidate model is generated, inherit the matching weight parameters from the original model. For parameters with completely identical shapes, load them directly; for parameters whose shapes have changed due to channel clipping, perform weight migration based on the retained channel index.

[0035] Furthermore, the proxy coarse screening does not perform full training on candidate models. Instead, it scores candidate models based on their computational cost, parameter count, inference latency, and shallow layer protection penalty relative to the original model, quickly eliminating clearly inferior candidate structures. The proxy coarse screening scoring function is expressed as:

[0036]

[0037] in, , , These represent the theoretical computational cost, parameter count, and inference latency of the candidate model, respectively. , , These represent the theoretical computational cost, number of parameters, and inference latency of the original model, respectively. This indicates a shallow protection penalty. , , , These represent the weight coefficients of each evaluation item.

[0038] Furthermore, the weight inheritance process can be represented as:

[0039] ;

[0040] in, Represents the weights of the original model. This represents the set of channel indices retained by the candidate model. This represents the weight inheritance function. This represents the weights inherited by the candidate model.

[0041] Furthermore, step 4 includes:

[0042] Step 4.1: Using candidate pruning vectors R as individuals in the population, the NSGA-II multi-objective genetic algorithm is used to search for pruning structures. After initializing the population, structural validity checks, proxy coarse screening, and quick eval fine screening are performed on each candidate individual in sequence to obtain the detection accuracy, theoretical computational cost, and actual inference latency of the candidate individuals.

[0043] Step 4.2: Set three pruning optimization objectives: maximize the detection accuracy of the candidate model; minimize the theoretical computational cost of the candidate model; minimize the actual inference latency of the candidate model. The objective function is expressed in minimization form: ;

[0044] Step 4.3: Perform non-dominated sorting on the candidate individuals in the population to obtain different Pareto levels; calculate the crowding distance for individuals within the same level to maintain the diversity of the Pareto front; then perform tournament selection based on the non-dominated sorting level and crowding distance to select the better individuals as parents; perform crossover and mutation operations on the parent individuals to generate new candidate pruning vectors.

[0045] The candidate individuals after crossover and mutation need to undergo legality correction to ensure that they meet the upper and lower limits of the channel retention ratio, shallow protection constraints, and channel coordination constraints of each topology pruning group. For individuals that do not meet the constraints, they are corrected to the allowable range, or the channel retention ratio of the relevant groups is readjusted according to the topology connection relationship to ensure that the generated candidate model structure is trainable and inferenceable.

[0046] Step 4.4: Merge the parent and offspring populations, recalculate the non-dominated sorting and crowding distance, and select the next generation population using an elite retention strategy; repeat the candidate structure generation, phased evaluation, non-dominated sorting, selection, crossover, mutation, and elite retention process until the maximum number of generations is reached, the Pareto front change is less than a preset threshold for multiple consecutive generations, or the preset search time limit is met, and finally obtain the Pareto optimal candidate pruning structure set;

[0047] Step 4.5: Select representative candidate structures from the Pareto front, including the candidate with the highest detection accuracy, the candidate with the lowest theoretical computational cost, the candidate with the lowest inference latency, and the comprehensive compromise candidate. The comprehensive compromise candidate is selected based on the distance of the candidate structure from the ideal point. The closer the candidate structure is to the ideal point, the better its overall performance in terms of detection accuracy, computational cost, and inference latency. The formula for selecting the comprehensive compromise candidate is as follows:

[0048] ;

[0049] in, Indicates the first The distance from each candidate structure to the ideal point; This indicates the normalized detection accuracy. This represents the theoretical computational cost after normalization. This represents the inference delay after normalization. (Choose) The smallest candidate structure is selected as the overall compromise candidate. The selected representative candidate structures are re-evaluated using complete training data and more training rounds to obtain more stable indicators such as detection accuracy, theoretical computational cost, parameter count, inference latency, and FPS. The final pruning structure is then determined based on actual deployment requirements.

[0050] Furthermore, step 5 includes the following steps:

[0051] Step 5.1: Based on the final pruning structure, perform structured channel pruning on the original YOLO model to generate the final lightweight YOLO detection model;

[0052] Step 5.2: Based on the channel retention index of the final pruned model, transfer the inheritable weights from the original model, and use the complete chip defect training set to formally fine-tune the final pruned model so that the model adapts to the pruned network structure and restores the detection accuracy.

[0053] Step 5.3: After the formal fine-tuning is completed, evaluate the final pruned model on the test set.

[0054] Furthermore, the topology pruning group includes one or more of the following: shallow high-resolution feature group, mid-level feature extraction group, deep semantic feature group, small target detection path group, medium target detection path group, and large target detection path group.

[0055] Furthermore, a constructibility check is performed on the candidate YOLO model structure to check whether the number of input channels of each feature fusion node meets the connection requirements, and whether the input feature scale and number of channels of the detection head meet the forward inference requirements. If the candidate model does not meet the above requirements, the channel retention ratio of the relevant topology pruning group is corrected according to the channel coordination constraint; denoted as: ;in, This represents the original YOLO model, where R represents the group-level pruning vector, and represents the pruning vector. The structure mapping function for generating candidate models, This represents the candidate pruning model.

[0056] Compared with the prior art, the present invention has at least the following beneficial effects:

[0057] (1) By dividing the topological pruning group, the layer-by-layer independent pruning is transformed into group-level collaborative pruning, which reduces the search dimension and reduces the risk of channel mismatch caused by layer-by-layer pruning in the YOLO multi-scale feature fusion path.

[0058] (2) By using shallow protection constraints, the edge, texture and location information required for the detection of small defects are preserved, thereby reducing the risk of missing small target defects on the chip.

[0059] (3) By combining proxy coarse screening and quick eval fine screening in a phased evaluation, the number of candidate pruning structures that require full training is reduced, and the pruning search efficiency is improved.

[0060] (4) By simultaneously considering detection accuracy, GFLOPs and actual inference latency, the final pruning structure not only pursues a reduction in theoretical computational load, but also takes into account the actual deployment acceleration effect.

[0061] (5) This method can adjust the number of pruning groups, retention ratio threshold and candidate evaluation strategy according to different YOLO network structures, defect categories and deployment devices, and has good adaptability. Attached Figure Description

[0062] Figure 1 Overall flowchart of topology-aware multi-objective structured pruning.

[0063] Figure 2 Group-level pruning coding and constraint correction diagram.

[0064] Figure 3 A schematic diagram of the topology pruning group partitioning in the YOLO model.

[0065] Figure 4 System functional module structure diagram. Detailed Implementation

[0066] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0067] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention.

[0068] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0069] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0070] like Figure 1As shown, this invention provides a lightweight method for YOLO chip defect detection models using topology-aware multi-objective structured pruning. The method mainly includes model and data acquisition, YOLO network topology analysis, topology pruning group partitioning, group-level pruning encoding, candidate structure validity correction, phased evaluation of candidate structures, NSGA-II multi-objective search, representative candidate review, and final pruned model generation and deployment evaluation. The data transfer relationship between each step is as follows: the original model and dataset are used to establish baseline indicators; the topology analysis results are used to generate topology pruning groups; group-level pruning vectors are used to generate candidate models; after phased evaluation, the candidate models enter multi-objective search; and finally, the lightweight detection model is determined by the Pareto candidate set. Specifically, it includes the following steps:

[0071] Step 1: Obtain the YOLO chip defect detection model to be pruned and the chip defect detection dataset, and use the chip defect detection dataset to establish the original model baseline.

[0072] Step 1.1: Obtain the YOLO chip defect detection model to be pruned, the original model weights, and the chip defect detection dataset.

[0073] More specifically, the YOLO chip defect detection model can be a YOLO series (such as YOLOv8, YOLOv10, YOLOv11) or a target detection model improved based on the YOLO series. The specific network version can be selected according to the chip defect detection task, the computing power of the deployed equipment, and the real-time requirements.

[0074] More specifically, the chip defect detection dataset undergoes preprocessing, including image size standardization, label format checking, category number checking, and the partitioning of training, validation, and test sets. To improve the model's ability to detect chip defects under different lighting conditions, defect morphologies, and backgrounds, data augmentation operations such as random scaling, flipping, brightness changes, and contrast adjustments can be performed on the training images.

[0075] More specifically, chip defect images are acquired to form a chip defect detection dataset, which is divided into training set, validation set and test set. Defect categories can include chip defect categories such as broken, crack, pollution, no_die, etc., or can be set to categories such as crack, breakage, contamination, missing, etc., depending on the actual object being detected.

[0076] Step 1.2: Based on the preprocessed chip defect detection dataset, the chip defect images are input into the original YOLO detection model for training, validation, and testing to obtain the baseline performance index of the unpruned model. The baseline index is used for normalized comparison of subsequent candidate pruned models, enabling different candidate structures to be evaluated under a unified standard.

[0077] The baseline performance metrics include detection accuracy, number of parameters, theoretical computational cost, model size, and average inference latency. The baseline metrics are expressed as follows:

[0078]

[0079] in, This indicates the detection accuracy of the original model. This represents the theoretical computational cost of the original model. This represents the number of parameters in the original model. This represents the average inference latency of the original model. This represents the original model size. Detection accuracy is preferably represented by mAP@0.5, theoretical computational cost is preferably represented by GFLOPs, and average inference latency is obtained by averaging multiple inference speed tests performed under specified hardware conditions.

[0080] In this application, the purpose of establishing the baseline of the original model is to provide a reference for the evaluation of subsequent candidate pruned structures. The computational cost, parameter count, and inference latency of all subsequent candidate models are normalized and compared with the original model, thereby avoiding the impact of inconsistent dimensions of different indicators on the evaluation of candidate structures.

[0081] Step 2: Analyze the network topology of the YOLO chip defect detection model to be pruned, divide it into topology pruning groups, and use the channel retention ratio of each topology pruning group as the group-level pruning vector; then calculate the number of channels that should be retained in each group after pruning; based on the candidate pruning vectors and the number of channels in each topology pruning group, perform structural mapping on the network structure corresponding to the original YOLO model to generate candidate pruning models; perform structural legality and constructibility checks on the candidate models; and correct the channel retention ratio of the relevant topology pruning groups according to channel coordination constraints.

[0082] Step 2.1: Read the network structure file of the YOLO chip defect detection model to be pruned, and analyze the connection relationships between the backbone network, neck network, and detection head in the model. Identify pruningable structures in the model, such as convolutional layers, feature extraction modules, feature fusion modules, and the pre-feature layer of the detection head. For structures with connection relationships such as Concat, Add, Upsample, and Detect, record their upstream and downstream channel dependencies as the basis for subsequent pruning constraints.

[0083] refer to Figure 3 In the YOLO model, the backbone network is used to extract shallow texture features and deep semantic features of the chip image, the neck network is used for multi-scale feature fusion, and the detection head is used to output the location and category of chip defect targets.

[0084] Step 2.2: As Figure 3 As shown, based on feature map scale, upstream and downstream connectivity, multi-scale feature fusion paths, and detection head input relationships, the YOLO model is divided into several topology pruning groups. These topology pruning groups may include one or more of the following: shallow high-resolution feature group, mid-level feature extraction group, deep semantic feature group, small target detection path group, medium target detection path group, and large target detection path group.

[0085] In a preferred embodiment, the YOLO chip defect detection model is divided into seven topology pruning groups, corresponding to the shallow backbone feature group, shallow small target path group, mid-layer feature group, deep semantic group, small target fusion group, mid-target fusion group, and large target fusion group, respectively. For different versions or improved structures of the YOLO model, the number and corresponding range of topology pruning groups can be increased, decreased, or merged according to the actual network connection relationship.

[0086] The shallow backbone feature group and shallow small target path are mainly used to preserve the edge, texture, and positional information required for detecting minute defects in the chip; the mid-layer feature group and deep semantic group are mainly used to extract higher-level semantic features; the small target fusion group, mid-target fusion group, and large target fusion group correspond to the feature fusion process of different scale detection branches. Through the above topological grouping, network layers or modules with connection dependencies can be pruned as a whole, avoiding the destruction of the YOLO multi-scale feature fusion structure by pruning layer by layer independently.

[0087] Shallow protection constraints are set for shallow backbone feature groups and shallow small target paths to reduce the risk of missing small defects; channel coordination constraints are set for connection positions such as Concat, Add and Detect to ensure that the feature paths at each scale after pruning can maintain channel matching and ensure that the model can be built and inferred normally.

[0088] Step 2.3: After completing the topology pruning group division, construct the group-level pruning vector using the channel retention ratio of each topology pruning group as the pruning search variable.

[0089] Suppose the YOLO model to be pruned is divided into K topological pruning groups, then the candidate pruning vector can be represented as:

[0090]

[0091] Where K represents the number of topologically pruned groups. This represents the channel retention ratio of the k-th topology pruning group. This represents the minimum channel retention rate for the k-th topology pruning group. This represents the highest channel retention rate in the k-th topology pruning group.

[0092] This invention employs a group-level pruning coding method of "one group, one variable," which groups multiple layers or modules with topological relationships into the same pruning group, allowing them to be structurally pruned according to a unified or coordinated retention ratio, rather than setting a pruning rate independently for each layer in the model. This reduces the search dimensionality of the genetic algorithm and reduces the multi-scale fusion path imbalance and channel mismatch problems caused by independent pruning layer by layer.

[0093] Step 2.4: Based on the channel retention ratio of each topology pruning group obtained in Step 2.3 and the original number of channels in the corresponding pruning group, calculate the number of channels that should be retained in each group after pruning.

[0094] For the calculated number of channels to be retained in each group after pruning, channel alignment is performed according to the network structure and deployment hardware requirements to ensure that the number of pruned channels is a preset integer multiple, while ensuring that structures with channel dependencies, such as Concat, Add, and Detect, can be connected normally. The channel alignment formula is expressed as:

[0095]

[0096] in, Indicates the first The number of channels in a topological pruning group before pruning. This indicates the number of channels after pruning. This represents a channel alignment function used to adjust the number of pruned channels to an integer multiple that satisfies network connectivity and hardware computation requirements.

[0097] When multiple network nodes participate in feature fusion, the number of channels in the relevant pruning groups is coordinated according to the corresponding topological dependencies to avoid input-output channel mismatch due to independent pruning of different branches.

[0098] Step 2.5: Based on the candidate pruning vectors obtained in Step 2.3 and the number of channels in each topology pruning group determined in Step 2.4, perform structure mapping on the network structure corresponding to the original YOLO model to generate candidate pruning models. Each candidate pruning vector corresponds to a candidate YOLO model structure. Therefore, the process of the genetic algorithm searching for pruning vectors essentially corresponds to searching for different lightweight YOLO network structures.

[0099] After candidate YOLO models are generated, structural legality and constructibility checks are performed. These checks include: whether the input and output channels of each feature fusion node in the candidate model meet the connectivity requirements; whether the feature branches corresponding to structures such as Concat and Add can be fused correctly; whether the number, scale, and number of channels of the feature maps input to the Detect head meet the requirements of the Detect head; and whether the candidate model can complete forward propagation correctly. If the candidate model does not meet the above requirements, the channel retention ratio of the relevant topology pruning groups is adjusted according to channel coordination constraints.

[0100]

[0101] in, This represents the original YOLO model, where R represents the group-level pruning vector, and represents the pruning vector. The structure mapping function for generating candidate models, This represents the candidate pruning model.

[0102] Step 3: Based on the dependence of chip micro-defects on shallow high-resolution features, channel retention constraints are set for the shallow protection group and small target detection related paths, and channel coordination correction is performed on candidate pruning structures; candidate pruning structures are evaluated in stages through proxy coarse screening and quick eval fine screening; Step 3.1: Considering that chip defects are usually small in size, have fine texture, and weak edge features, the target detection model has a strong dependence on shallow high-resolution features. To avoid over-compressing shallow features during pruning, the shallow backbone feature path, shallow small target feature path, and topology pruning group related to high-resolution feature fusion are determined as the shallow protection group, and a minimum channel retention ratio is set for the shallow protection group.

[0103] In a preferred embodiment, the minimum channel retention ratio for the shallow protection group is set to 0.85, meaning that the corresponding shallow protection group is not allowed to be over-compressed. When the channel retention ratio of the corresponding shallow protection group in a candidate pruning vector is lower than the minimum allowable value, the candidate pruning vector is directly corrected to restore it to the allowable range. The minimum channel retention ratio can be adjusted according to the chip defect size, image resolution, YOLO model structure, and computing power of the deployed device, and is not limited to 0.85.

[0104] Step 3.2: Based on meeting the minimum shallow channel retention ratio in Step 3.1, a shallow soft protection mechanism is further implemented. When the channel retention ratio of the shallow protection group, although not lower than the minimum allowable value, is lower than the preset shallow soft protection target value, a shallow protection penalty is added during the candidate structure evaluation process. As the channel retention ratio of the shallow protection group decreases, the penalty on the corresponding candidate structure increases, thereby ensuring that the genetic search process, while meeting the requirements of model lightweighting, tends to retain high-resolution shallow features that play an important role in the detection of small defects in the chip.

[0105] Step 3.3: As Figure 1 As shown, for candidate pruned structures generated by the genetic algorithm and whose legitimacy has been corrected, a proxy coarse screening is first performed. Using the original model baseline established in step 1, the computational cost, parameter count, and inference latency of the candidate models are normalized, and the proxy evaluation results of the candidate structures are obtained by combining this with shallow protection penalties. Candidate models that clearly fail to meet the lightweight requirements, have excessive shallow key feature compression, or have poor overall evaluation results are directly eliminated at this stage, preventing them from entering the subsequent quick eval stage, which requires short training rounds.

[0106] In this embodiment, the proxy coarse screening does not perform full training on candidate models. Instead, it scores candidate models based on their computational cost, parameter count, inference latency, and shallow protection penalty relative to the original model, quickly eliminating clearly inferior candidate structures. This step reduces the number of candidates that need to be trained and validated in the subsequent quick eval stage, thereby lowering the overall search cost. The proxy coarse screening scoring function is expressed as:

[0107] in, , , These represent the candidate model's GFLOPs, parameter count, and inference latency, respectively. , , These represent the original model's GFLOPs, number of parameters, and inference latency, respectively. This indicates a shallow protection penalty. , , , These represent the weight coefficients of each evaluation item.

[0108] Step 3.4: Perform quick eval refinement on the candidate models selected through proxy coarse screening. First, generate corresponding model configuration files based on the candidate pruned vectors and calculate the number of parameters, GFLOPs, and model size of the candidate models. Then, perform multiple inference speed tests in a specified hardware environment to obtain the average inference latency of the candidate models. Next, inherit dimension-matched weight parameters from the original model weights, initialize parameters that cannot be directly inherited, and perform short training epochs using a portion of the training data. Finally, obtain the detection accuracy of the candidate models on the validation set. Through these steps, a balance can be achieved between computational cost and evaluation reliability.

[0109] Step 3.5: Inherit the matching parameters of the candidate model from the original model weights. For parameters with completely identical shapes, directly inherit the original model weights; for parameters whose shapes have changed due to channel pruning, perform weight transfer based on the preserved channel index. Weight inheritance can reduce the instability caused by training candidate models from random initialization and improve the comparability of evaluation results between candidate structures.

[0110] After weight inheritance is completed, the candidate model is trained for a short period of time using a portion of the training data, and the detection accuracy is calculated on the validation set to obtain the evaluation results such as mAP@0.5, GFLOPs, and inference latency of the candidate structure.

[0111] The weight inheritance process is represented as follows:

[0112]

[0113] in, Represents the weights of the original model. This represents the set of channel indices retained by the candidate model. This represents the weight inheritance function. This represents the weights inherited by the candidate model.

[0114] Step 4: Use the NSGA-II multi-objective genetic algorithm to search for the final pruning structure that balances detection accuracy, GFLOPs, and actual inference latency.

[0115] Step 4.1: Using the candidate pruning vectors constructed in Step 2.3 as individuals in the NSGA-II multi-objective genetic algorithm, an initial candidate pruning structure is generated according to the preset population size. Each candidate individual sequentially performs candidate structure generation, structure validity check, proxy coarse screening, and quick eval fine screening, and obtains the detection accuracy, GFLOPs, and actual inference latency of the corresponding candidate model.

[0116] Step 4.2: Establish multi-objective pruning optimization objectives. Addressing the simultaneous requirements of accuracy, computational complexity, and inference speed in the practical deployment of chip defect detection models, this invention sets three pruning optimization objectives: maximizing the detection accuracy of candidate models; minimizing the GFLOPs of candidate models; and minimizing the actual inference latency of candidate models. The objective function is expressed in minimization form:

[0117]

[0118] in, This indicates the detection accuracy of the candidate model, preferably mAP50; Indicates the theoretical computational cost of candidate models, with GFLOPs being the preferred choice; This indicates the actual inference latency of the candidate model. By considering the above three objectives simultaneously, this invention avoids the problem that simply compressing GFLOPs does not significantly improve the actual inference speed, making the pruning structure more in line with actual deployment requirements.

[0119] Step 4.3: Perform non-dominated sorting, selection, crossover, and mutation. Based on the multiple optimization objectives defined in Step 4.2, perform non-dominated sorting on candidate individuals in the population, dividing the candidate pruning structure into different Pareto levels. For candidate individuals at the same Pareto level, calculate their crowding distance to maintain the distribution diversity of different pruning schemes on the Pareto front. Subsequently, perform tournament selection based on the candidate individuals' non-dominated sorting level and crowding distance, selecting the better individuals as parents, and performing crossover and mutation operations on the parent pruning vectors to generate new candidate pruning vectors.

[0120] Candidate individuals resulting from crossover and mutation require legitimacy adjustments to ensure they meet the upper and lower limits of channel retention ratios, shallow protection constraints, and channel coordination constraints for each topology pruning group. For candidate individuals that do not meet these constraints, the corresponding channel retention ratio is adjusted to the allowable range, or the retention ratios of relevant pruning groups are jointly adjusted based on network topology dependencies to ensure that the generated candidate model structure is trainable and inference-compatible.

[0121] Step 4.4: Merge the parent and offspring populations, recalculate the non-dominated sorting and crowding distance, and select the next generation population using an elite retention strategy. Repeat the candidate structure generation, phased evaluation, non-dominated sorting, selection, crossover, mutation, and elite retention processes until the maximum number of generations is reached, the Pareto front change is less than a preset threshold for multiple consecutive generations, or the preset search time limit is met, ultimately obtaining the Pareto optimal candidate pruned structure set.

[0122] Step 4.5: Select representative candidate structures from the Pareto front, including the candidate with the highest detection accuracy, the candidate with the lowest GFLOPs, the candidate with the lowest inference latency, and the comprehensive compromise candidate. The comprehensive compromise candidate is selected based on the distance of the candidate structure from the ideal point. The closer the candidate structure is to the ideal point, the better its overall performance in terms of detection accuracy, computational cost, and inference latency.

[0123] The formula for a comprehensive compromise candidate selection is expressed as follows:

[0124]

[0125] in, Indicates the first The distance from each candidate structure to the ideal point; This indicates the normalized detection accuracy. Represents the normalized GFLOPs; This represents the inference delay after normalization. (Choose) The smallest candidate structure is selected as the overall compromise candidate. The selected representative candidate structures are re-evaluated using complete training data and more training rounds to obtain more stable indicators such as detection accuracy, GFLOPs, parameter count, inference latency, and FPS. The final pruning structure is then determined based on actual deployment requirements.

[0126] Step 5: Perform structured channel pruning on the original YOLO model based on the final pruning structure to obtain a lightweight YOLO chip defect detection model suitable for deployment.

[0127] Step 5.1: Based on the final pruning structure, perform structured channel pruning on the original YOLO model to generate the final lightweight YOLO detection model. The structured channel pruning is performed by deleting convolutional channels, module channels, or detection path channels, and the pruned model maintains a regular network structure.

[0128] Step 5.2: Based on the channel-preserving index of the final pruned model, transfer the inheritable weights from the original model, and formally fine-tune the final pruned model using a complete chip defect training set to adapt the model to the pruned network structure and restore detection accuracy. After formal fine-tuning, evaluate the final pruned model on the test set to obtain metrics such as detection accuracy, number of parameters, theoretical computational cost, model size, inference latency, and FPS.

[0129] Step 5.3: Calculate the parameter compression ratio, theoretical computational compression ratio, and inference speedup ratio of the final pruned model relative to the original model:

[0130]

[0131] in, Indicates the compression ratio parameter. This represents the compression ratio of theoretical calculations. Indicates the reasoning speedup ratio; , , These represent the number of parameters, theoretical computational cost, and inference latency of the final pruning model, respectively.

[0132] The final pruned model is exported as a model file suitable for deployment environments and deployed to industrial control computers, embedded devices, edge computing terminals, or production line inspection equipment according to actual application needs, thereby completing YOLO model topology-aware multi-objective structured pruning for chip defect detection tasks.

[0133] Based on the above method, a YOLO model for chip defect detection can be constructed, and the model can be applied to chip defect detection, taking the chip image to be detected as input and using the model to directly output the recognition result.

[0134] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A lightweight method for YOLO chip defect detection model utilizing topology-aware multi-objective structured pruning, characterized in that, Includes the following steps: Step 1: Using the original YOLO detection model to be pruned, the original model weights, and the chip defect detection dataset as input, establish the original model baseline; Step 2: Analyze the topological connection relationship between the backbone network, neck network and detection head, divide the structure with channel dependency relationship or feature fusion relationship into several topological pruning groups, and use the channel retention ratio of each topological pruning group as the group-level pruning coding variable. Step 3: Based on the dependence of chip micro-defects on shallow high-resolution features, channel retention constraints are set for shallow protection groups and small target detection related paths, and channel coordination correction is performed on candidate pruning structures; then, candidate pruning structures are evaluated in stages through proxy coarse screening and quick eval fine screening. Step 4: Use the NSGA-II multi-objective genetic algorithm to search for the final pruning structure that balances detection accuracy, theoretical computational cost, and actual inference latency. Step 5: Perform structured channel pruning on the original YOLO detection model based on the final pruning structure to obtain a lightweight YOLO chip defect detection model suitable for deployment.

2. The lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Obtain the original YOLO detection model, the original model weights, and the chip defect detection dataset; the chip defect detection dataset includes chip defect images and defect type labels; Step 1.2: Train, validate, and test the original YOLO detection model using the chip defect detection dataset to obtain the baseline performance metrics of the unpruned model. These baseline performance metrics include detection accuracy, number of parameters, theoretical computational cost, model size, and actual inference latency; expressed as: ； in, This indicates the detection accuracy of the original model. This represents the theoretical computational cost of the original model. This represents the number of parameters in the original model. This represents the average inference latency of the original model. This indicates the size of the original model.

3. The lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: Read the network structure file of the YOLO model to be pruned, analyze the connection relationship between the backbone network, neck network and detector head in the model, and identify the pruning structures in the model such as convolutional layers, feature extraction modules, feature fusion modules and front feature layers of the detector head; Step 2.2: Based on the feature map scale, upstream and downstream connectivity, multi-scale feature fusion path, and detection head input relationship, divide the model to be pruned into K topology pruning groups; Step 2.3: Using the channel retention ratio of each topological pruning group as the encoding variable for the individual in the genetic algorithm, construct the group-level pruning vector, represented as: ; This represents the channel retention ratio of the k-th topology pruning group. , This represents the minimum channel retention rate for the k-th topology pruning group. This represents the highest channel retention rate in the k-th topology pruning group; Step 2.4: Based on the channel retention ratio of each topology pruning group, calculate the number of channels that should be retained in each group after pruning; and align the calculated channel numbers. The channel alignment formula is expressed as: ;in, Indicates the first The number of channels in a topological pruning group before pruning. This indicates the number of channels after pruning. This represents the channel alignment function, used to adjust the number of pruned channels to an integer multiple that satisfies network connectivity and hardware computation requirements; Step 2.5: Based on the candidate pruning vectors and the number of channels in each topology pruning group, perform structural mapping on the network structure corresponding to the original YOLO model to generate candidate pruning models, and check the structural legality and constructibility of the candidate pruning models.

4. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning as described in claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Based on the characteristics of chip defect targets, the shallow backbone feature path, the shallow small target feature path, and the topology pruning group related to high-resolution feature fusion are identified as the shallow protection group, and a minimum channel retention ratio is set for the shallow protection group; the shallow protection constraint is expressed as: ;in, This represents a set of shallow protection groups. Indicates the minimum retention ratio in the shallow layer. For the k-th topological pruning group, Represents topological pruning groups The corresponding channel retention ratio; Step 3.2: Based on meeting the minimum shallow channel retention ratio, a shallow soft protection penalty term is set during the candidate structure screening process. When the channel retention ratio of the shallow protection group is lower than the preset soft protection target value, a penalty is applied to the candidate pruned structure, so that the genetic search process prioritizes the retention of shallow high-resolution feature paths that are important for the detection of minute defects. The shallow protection penalty term is expressed as follows: ;in, This indicates a shallow protection penalty. This represents a set of shallow protection groups. This represents the target value for shallow soft protection. Indicates the first The channel retention ratio of each topology pruning group; when Below In such cases, the candidate structure is penalized accordingly; Step 3.3: Perform a proxy coarse screening on the candidate pruning structures generated by the genetic algorithm. Step 3.4: Perform quick eval fine screening on the candidate pruned structures that have passed the proxy coarse screening; Step 3.5: After the candidate model is generated, inherit the matching weight parameters from the original model. For parameters with completely identical shapes, load them directly; for parameters whose shapes have changed due to channel clipping, perform weight migration based on the retained channel index.

5. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning as described in claim 4, characterized in that, The proxy coarse screening does not perform full training on candidate models. Instead, it scores candidate models based on their computational cost, parameter count, inference latency, and shallow layer protection penalty relative to the original model, quickly eliminating clearly inferior candidate structures. The proxy coarse screening scoring function is expressed as: in, , , These represent the theoretical computational cost, parameter count, and inference latency of the candidate model, respectively. , , These represent the theoretical computational cost, number of parameters, and inference latency of the original model, respectively. This indicates a shallow protection penalty. , , , These represent the weight coefficients of each evaluation item.

6. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning according to claim 4, characterized in that, The weight inheritance process is represented as follows: ;in, Represents the weights of the original model. This represents the set of channel indices retained by the candidate model. This represents the weight inheritance function. This represents the weights inherited by the candidate model.

7. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning as described in claim 1, characterized in that, Step 4 includes: Step 4.1: Using candidate pruning vectors R as individuals in the population, the NSGA-II multi-objective genetic algorithm is used to search for pruning structures. After initializing the population, structural validity checks, proxy coarse screening, and quickeval fine screening are performed on each candidate individual in sequence to obtain the detection accuracy, theoretical computational cost, and actual inference latency of the candidate individuals. Step 4.2: Set three pruning optimization objectives: maximize the detection accuracy of the candidate model; minimize the theoretical computational cost of the candidate model; minimize the actual inference latency of the candidate model. The objective function is expressed in minimization form: ; Step 4.3: Perform non-dominated sorting on the candidate individuals in the population to obtain different Pareto levels; calculate the crowding distance for individuals within the same level to maintain the diversity of the Pareto front; then perform tournament selection based on the non-dominated sorting level and crowding distance to select the better individuals as parents; perform crossover and mutation operations on the parent individuals to generate new candidate pruning vectors. The candidate individuals after crossover and mutation need to undergo legality correction to ensure that they meet the upper and lower limits of the channel retention ratio, shallow protection constraints, and channel coordination constraints of each topology pruning group. For individuals that do not meet the constraints, they are corrected to the allowable range, or the channel retention ratio of the relevant groups is readjusted according to the topology connection relationship to ensure that the generated candidate model structure is trainable and inferenceable. Step 4.4: Merge the parent and offspring populations, recalculate the non-dominated sorting and crowding distance, and select the next generation population using an elite retention strategy; repeat the candidate structure generation, phased evaluation, non-dominated sorting, selection, crossover, mutation, and elite retention process until the maximum number of generations is reached, the Pareto front change is less than a preset threshold for multiple consecutive generations, or the preset search time limit is met, and finally obtain the Pareto optimal candidate pruning structure set; Step 4.5: Select representative candidate structures from the Pareto front, including the candidate with the highest detection accuracy, the candidate with the lowest theoretical computational cost, the candidate with the lowest inference latency, and the comprehensive compromise candidate. The comprehensive compromise candidate is selected based on the distance of the candidate structure from the ideal point. The closer the candidate structure is to the ideal point, the better its overall performance in terms of detection accuracy, computational cost, and inference latency. The formula for selecting the comprehensive compromise candidate is as follows: ; in, Indicates the first The distance from each candidate structure to the ideal point; This indicates the normalized detection accuracy. This represents the theoretical computational cost after normalization. Indicates the normalized inference delay; selection The smallest candidate structure is selected as the overall compromise candidate. The selected representative candidate structures are then re-evaluated using complete training data and more training rounds to obtain more stable indicators such as detection accuracy, theoretical computational cost, parameter count, inference latency, and FPS. The final pruning structure is then determined based on actual deployment requirements.

8. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning according to claim 1, characterized in that, Step 5 includes the following steps: Step 5.1: Based on the final pruning structure, perform structured channel pruning on the original YOLO model to generate the final lightweight YOLO detection model; Step 5.2: Based on the channel retention index of the final pruned model, transfer the inheritable weights from the original model, and use the complete chip defect training set to formally fine-tune the final pruned model so that the model adapts to the pruned network structure and restores the detection accuracy. Step 5.3: After the formal fine-tuning is completed, evaluate the final pruned model on the test set.

9. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning as described in claim 1, characterized in that, The topology pruning group includes one or more of the following: shallow high-resolution feature group, mid-level feature extraction group, deep semantic feature group, small target detection path group, medium target detection path group, and large target detection path group.

10. A lightweight method for YOLO chip defect detection model using topology-aware multi-objective structured pruning according to claim 1, characterized in that, The constructability of candidate YOLO model structures is checked, including whether the number of input channels of each feature fusion node meets the connection requirements, and whether the input feature scale and number of channels of the detection head meet the forward inference requirements. If the candidate model does not meet the above requirements, the channel retention ratio of the relevant topology pruning group will be corrected according to the channel coordination constraints. Recorded as: ;in, This represents the original YOLO model, where R represents the group-level pruning vector, and represents the pruning vector. The structure mapping function for generating candidate models, This represents the candidate pruning model.

Images