Chip needle scratch precise detection method based on semi-supervised self-evolution

Abstract

The application discloses a chip needle mark precise detection method based on semi-supervised self-evolution, constructs a chip needle mark basic detection network model, adopts source domain sufficient supervised needle mark samples to train the chip needle mark basic detection network model, and selects stable teacher model snapshots in different training stages to construct a joint teacher model group; a plurality of teacher models in the joint teacher model group are used for cooperative reasoning on a plurality of target domain unlabeled samples, and needle mark targets in the target domain unlabeled samples are divided into a high-reliability sample set and a drift sample set; and the chip needle mark basic detection network model is supervised and trained again based on the high-reliability sample set and the drift sample set, so that an updated student model is obtained; after the updated student model is obtained, model parameters of the joint teacher model group are iteratively updated in a differential EMA mode until an optimal student model is obtained; and the optimal student model is used for needle mark detection on target domain chip images, so that needle mark detection results are obtained.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method for accurate detection of chip pin marks based on semi-supervised self-evolution. Background Technology

[0002] In the integrated circuit manufacturing process, wafer probing (CP) is a core step in packaging and testing. Probe cards use tiny probes to contact the chip pads and apply electrical signals; the resulting pin marks contain crucial process information. The depth and shape of these pin marks directly reflect contact quality: too shallow a mark may lead to the rejection of good products, while too deep a mark may damage the passivation layer and introduce reliability issues. Therefore, high-precision automated detection of pin marks is essential for monitoring probe card status and ensuring chip yield. As advanced process nodes continue to shrink and chip pad sizes decrease, pin marks under microscopic imaging exhibit characteristics such as extremely small scale, weak edges, dense distribution, and susceptibility to background texture interference. Traditional microscopic visual inspection methods relying on manual experience are no longer sufficient to meet the real-time, repeatability, and objectivity requirements of high-speed production lines. Existing pin mark detection methods based on traditional image processing or fully supervised depth networks typically rely heavily on known operating conditions and sufficient labeled data. In actual production lines, changes in probe station status, chip type, illumination angle, or imaging system can significantly alter the brightness distribution, background texture, and noise patterns of the image. This significant domain shift caused by changes in the physical environment reduces the generalization ability of static detection models trained under old operating conditions in new production lines, leading to performance degradation issues such as missed detections and false detections of needle marks.

[0003] Meanwhile, while existing teacher-student self-training methods can utilize unlabeled samples, their direct application to high-precision chip pin mark detection still has the following shortcomings: First, pin marks are tiny targets in industrial microscopic images, and even slight positioning deviations can lead to pseudo-label distortion, making it easy for a single teacher model to accumulate noise during continuous iteration; Second, existing sample selection mechanisms usually discard low-confidence samples directly, while these samples often contain lighting, texture, and noise distribution information unique to the target domain, and simply discarding them is not conducive to the model fully learning the characteristics of new production line data; Finally, relying solely on pseudo-label supervision is insufficient to eliminate the differences in multi-order statistical features between different probe stations, and the model's generalization ability to unknown working conditions remains limited. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a chip pin mark accurate detection method based on semi-supervised self-evolution. The method constructs a basic chip pin mark detection network model and constructs a joint teacher model group to divide the unlabeled samples in the target domain. The student model of the basic chip pin mark detection network model is optimized by using the divided sample set, so that the model can continuously self-update under the condition that the target domain is completely unlabeled by humans.

[0005] Technical Solution: To achieve the above objectives, the present invention provides a method for precise detection of chip pin marks based on semi-supervised self-evolution, comprising the following steps:

[0006] Step 1: Introduce the FEM feature enhancement module and SCRU spatial-channel reconstruction unit to construct a basic chip pin mark detection network model. Train the basic chip pin mark detection network model with sufficient supervised pin mark samples in the source domain to obtain the initial student model. During the source domain training process, select stable teacher model snapshots at different training stages and construct a joint teacher model group from the stable teacher model snapshots at different training stages.

[0007] Step 2: Using several teacher models in the joint teacher model group, collaborative reasoning is performed on the unlabeled samples in the target domain to obtain the average detection confidence and cognitive dissonance of each pin mark target in the unlabeled samples of the target domain; based on the average detection confidence and cognitive dissonance of each pin mark target, the pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set, a drift sample set, and a low-quality noise sample set.

[0008] Step 3: Use the highly reliable sample set to generate pseudo-labels for supervised learning and construct the target domain pseudo-label loss; use the drift sample set for multi-scale MMD and CMD feature alignment between the source and target domains and construct the feature alignment loss; weight and fuse the target domain pseudo-label loss, feature alignment loss, and source domain supervised loss to obtain the total loss, and then conduct supervised training on the initial student model again. During the supervised training process, the initial student model performs backpropagation on the total loss to update the model parameters, resulting in an updated student model.

[0009] Step 4: After obtaining the updated student model, iteratively update the model parameters of the joint teacher model group using the differentiated EMA method until the maximum number of iterations is reached to obtain the optimal student model; use the optimal student model to perform pin mark detection on the target domain chip image to obtain the pin mark detection results of the target domain chip image.

[0010] Furthermore, the chip pin mark detection network model includes a backbone network, a neck network, and a detection output network. The preprocessed input image is input into the backbone network, which performs multi-level downsampling on the input image to gradually obtain multiple hierarchical feature maps at different scales. These hierarchical feature maps at different scales are then input into the neck network, where an FEM feature enhancement module and a SCRU spatial-channel reconstruction unit are introduced to fuse and enhance the hierarchical feature maps at different scales, resulting in multiple feature maps at different scales. Finally, these feature maps at different scales are input into multiple detection heads at different resolutions in the detection output network to obtain the pin mark detection results for the input image.

[0011] Furthermore, the backbone network includes an Input layer, a Stem layer, a first SRFE layer, a first Conv convolutional layer, a second SRFE layer, a second Conv convolutional layer, a third SRFE layer, a third Conv convolutional layer, a fourth SRFE layer, and an SPPF+C2PSA layer. These layers are sequentially connected in series to form a linear network structure. The input image is input to the Input layer. The first SRFE layer outputs a first-level feature map, the second SRFE layer outputs a second-level feature map, the third SRFE layer outputs a third-level feature map, and the SPPF+C2PSA layer outputs a fourth-level feature map.

[0012] Furthermore, the neck network includes a feature enhancement module and a spatial-channel reconstruction module; the feature enhancement module includes a first FEM feature enhancement layer, a second FEM feature enhancement layer, a third FEM feature enhancement layer, a fourth FEM feature enhancement layer, a first fusion layer, a second fusion layer, a third fusion layer, and a fourth fusion layer; the first-level feature map, the second-level feature map, the third-level feature map, and the fourth-level feature map are respectively input into the first FEM feature enhancement layer, the second FEM feature enhancement layer, the third FEM feature enhancement layer, and the fourth FEM feature enhancement layer for feature enhancement, respectively, to obtain the first feature map, the second feature map, the third feature map, and the fourth feature map; the feature maps are then processed by average pooling. The third and fourth feature maps are both input into the fourth fusion layer for fusion to obtain the fourth fusion feature map; the second feature map after average pooling, the fourth feature map after upsampling, the fourth fusion feature map after upsampling, and the third feature map are all input into the third fusion layer for fusion to obtain the third fusion feature map; the first feature map after average pooling, the third feature map after upsampling, the third fusion feature map after upsampling, and the second feature map are all input into the second fusion layer for fusion to obtain the second fusion feature map; the second feature map after upsampling, the second fusion feature map after upsampling, and the first feature map are all input into the first fusion layer for fusion to obtain the first fusion feature map.

[0013] Furthermore, the spatial-channel reconstruction module includes a first C3K2 layer, a second C3K2 layer, a third C3K2 layer, a fourth C3K2 layer, a first SCRU spatial-channel reconstruction unit, a second SCRU spatial-channel reconstruction unit, a third SCRU spatial-channel reconstruction unit, and a fourth SCRU spatial-channel reconstruction unit. The first fused feature map, the second fused feature map, the third fused feature map, and the fourth fused feature map are respectively input into the first C3K2 layer, the second C3K2 layer, the third C3K2 layer, and the fourth C3K2 layer for convolution operations, obtaining the first convolution output, the second convolution output, the third convolution output, and the fourth convolution output, respectively. The first convolution output and the upsampled second convolution output are both input into the first SCRU spatial-channel reconstruction unit for reconstruction, obtaining the first SCRU spatial-channel reconstruction unit. The first convolutional output after average pooling, the third convolutional output after upsampling, the first feature map at different scales after average pooling, and the second convolutional output are all input into the second SCRU spatial-channel reconstruction unit for reconstruction to obtain the second feature map at different scales. The second convolutional output after average pooling, the fourth convolutional output after upsampling, the second feature map at different scales after average pooling, and the third convolutional output are all input into the third SCRU spatial-channel reconstruction unit for reconstruction to obtain the third feature map at different scales. The third convolutional output after average pooling, the third feature map at different scales after average pooling, and the fourth convolutional output are all input into the fourth SCRU spatial-channel reconstruction unit for reconstruction to obtain the fourth feature map at different scales.

[0014] Furthermore, in step two, several teacher models in the joint teacher model group are used to perform collaborative reasoning on the unlabeled samples in the target domain to obtain the average detection confidence and cognitive divergence of each pinhole target in the unlabeled samples of the target domain. It is assumed that there are K teacher models in the joint teacher model group. The joint teacher model group performs forward reasoning on the unlabeled samples in the target domain respectively, and constructs a deduplicated global candidate set Ω through spatial fusion. For the i-th pinhole target in the global candidate set Ω, the prediction probability of the pinhole target by the K teacher models in the joint teacher model group is extracted as the detection confidence of the pinhole target by the K teacher models, and the average detection confidence C of the pinhole target is calculated. i The calculation process is as follows:

[0015]

[0016] In the formula, Let be the confidence level of the k-th teacher model for the i-th needle mark target.

[0017] Furthermore, an entropy-based information decomposition mechanism is introduced to define the expected value of the classification entropy of a single teacher model as the uncertainty of the single teacher model's prediction, and to calculate the Shannon entropy of the average detection confidence of each pinprick target as the total uncertainty of the joint prediction for each pinprick target; the difference between the total uncertainty of the joint prediction for each pinprick target and the uncertainty of the single teacher model's prediction is defined as the cognitive divergence degree U of that pinprick target. dis,i The calculation is as follows:

[0018]

[0019]

[0020]

[0021] In the formula, U dis,i H represents the cognitive divergence of the i-th needle mark target. cls,i For the uncertainty of a single teacher model prediction for the i-th needle mark target, H pre,i The total uncertainty in the joint prediction of the i-th needle mark target.

[0022] Furthermore, based on the average detection confidence and cognitive divergence of each pin mark target, the pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set S. reliable Drift sample set S drift and low-quality noisy sample set S noise The calculation process is as follows:

[0023]

[0024]

[0025]

[0026] In the formula, C high and C low U represents the high threshold and low threshold of the average detection confidence, respectively. high and U low S represents the high threshold and low threshold of cognitive disagreement, respectively; reliable For a highly reliable sample set, S drift For the drift sample set, S noise Ω represents the set of low-quality, noisy samples, and Ω represents the global candidate set among the unlabeled samples in the target domain.

[0027] Furthermore, in step four, after obtaining the updated student model, the model parameters of the joint teacher model group are iteratively updated using the differentiated EMA method until the maximum number of iterations is reached, thus obtaining the optimal student model.

[0028] Let the parameters of the k-th teacher model at time t be... The parameters of the student model at time t+1 are The iterative update of the k-th teacher model is as follows:

[0029]

[0030] In the formula, α k Let be the momentum coefficient corresponding to the k-th teacher model, and K be the number of teacher models in the joint teacher model group.

[0031] Beneficial Effects: This invention provides a semi-supervised self-evolutionary chip pin mark detection method. It constructs a basic chip pin mark detection network model and introduces a structurally reparameterized feature extraction module (SRFE) into the backbone network. This enhances the joint modeling capability of fine-grained texture, edge grayscale changes, and deep semantic information of pin marks, while maintaining feature extraction accuracy and reducing inference computational overhead during the inference stage. The neck network introduces a feature enhancement module (FEM) and a SCRU spatial-channel reconstruction unit to mitigate the problem of small pin mark features being easily submerged by large-scale background information under conventional equal-weighted fusion methods, thus improving the accuracy of dense small-target features. The effective characterization capability of the target can reduce the interference of complex chip background, noise texture and pseudo response on the detection results; it divides the unlabeled samples in the target domain into a high-reliability sample set, a drift sample set and a low-quality noise sample set. The high-reliability sample set can provide stable semantic supervision for the target domain, while the drift sample set retains the distribution difference information of the new production line in terms of illumination, texture and noise, thereby avoiding the simple discarding of low-confidence but high-value samples; it can improve the utilization efficiency of unlabeled target domain data; and it can enable the target domain to continuously update itself under the condition of no manual annotation, so as to meet the rapid deployment needs of the new production line. Attached Figure Description

[0032] Figure 1 This is a semi-supervised self-evolutionary method for precise detection of chip pin marks.

[0033] Figure 2 This is a structural diagram of the chip pin mark detection network model. Detailed Implementation

[0034] The invention will now be further described with reference to the accompanying drawings.

[0035] like Figure 1 As shown, a method for accurate detection of chip pin marks based on semi-supervised self-evolution includes the following steps:

[0036] Step 1: Introduce the FEM feature enhancement module and SCRU spatial-channel reconstruction unit to construct a basic chip pin mark detection network model. Train the basic chip pin mark detection network model with sufficient supervised pin mark samples in the source domain to obtain the initial student model. During the source domain training process, select stable teacher model snapshots at different training stages and construct a joint teacher model group from the stable teacher model snapshots at different training stages.

[0037] Step 2: Using several teacher models in the joint teacher model group, collaborative reasoning is performed on the unlabeled samples in the target domain to obtain the average detection confidence and cognitive dissonance of each pin mark target in the unlabeled samples of the target domain; based on the average detection confidence and cognitive dissonance of each pin mark target, the pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set, a drift sample set, and a low-quality noise sample set.

[0038] Step 3: Use the highly reliable sample set to generate pseudo-labels for supervised learning and construct the target domain pseudo-label loss; use the drift sample set for multi-scale MMD and CMD feature alignment between the source and target domains and construct the feature alignment loss; weight and fuse the target domain pseudo-label loss, feature alignment loss, and source domain supervised loss to obtain the total loss, and then conduct supervised training on the initial student model again. During the supervised training process, the initial student model performs backpropagation on the total loss to update the model parameters, resulting in an updated student model.

[0039] Step 4: After obtaining the updated student model, iteratively update the model parameters of the joint teacher model group using the differentiated EMA method, and return to Steps 2 and 3 to update the student model until the maximum number of iterations is reached to obtain the optimal student model, which can continuously self-update under the condition that the target domain is completely unannotated; use the optimal student model to perform pin mark detection on the target domain chip image to obtain the pin mark detection results of the target domain chip image.

[0040] The above steps correspond to the construction of basic detection capabilities, the diversion of unlabeled samples in the target domain, and the collaborative iterative update by teachers and students, thus forming a closed-loop self-evolutionary framework of "joint teacher evaluation - instance-level diversion - student optimization - teacher rewrite". It can further distinguish unlabeled samples in the target domain into semantic supervision samples and distribution correction samples, so that highly reliable samples can play a stable supervision role and drifting samples can play a cross-domain alignment role, thereby improving the utilization rate of target domain data without introducing additional manual annotation.

[0041] The stable teacher model snapshots from different training stages are obtained by saving and filtering them in stages during the supervised training process in the source domain. Specifically, the chip pin mark detection network model is first trained using labeled pin mark samples from the source domain. Then, candidate model parameters are saved at different iteration stages of the source domain training according to a preset period. Next, based on the detection performance and convergence stability on the source domain validation set, several stable model parameters are selected from the candidate models at each stage as stable teacher model snapshots. Finally, the stable teacher model snapshots from different training stages are combined into a joint teacher model group for collaborative inference in the target domain. By using the joint teacher model group for collaborative inference in the target domain, the random bias of a single teacher model in the prediction of the target domain can be reduced.

[0042] like Figure 2 As shown, the chip pin mark detection network model belongs to a single-stage improved YOLO architecture. Addressing the characteristics of extremely small scale, dense distribution, and strong background interference of chip pin marks, the backbone network for feature extraction, the neck network for feature fusion, and the detection output network are jointly optimized to obtain the chip pin mark detection network model. This model is used as the foundation detection network for the subsequent teacher-student self-evolutionary framework. The chip pin mark detection network model includes a backbone network, a neck network, and a detection output network. The preprocessed input image is input into the backbone network, which performs multi-level downsampling to gradually obtain multiple hierarchical feature maps at different scales. These hierarchical feature maps are then input into the neck network, where an FEM feature enhancement module and a SCRU spatial-channel reconstruction unit are introduced to fuse and enhance the hierarchical feature maps at different scales, resulting in multiple feature maps at different scales. These feature maps at different scales are then input into multiple detection heads of different resolutions in the detection output network to obtain the pin mark detection results for the input image.

[0043] The backbone network comprises an Input layer, a Stem layer, a first SRFE layer, a first Conv convolutional layer, a second SRFE layer, a second Conv convolutional layer, a third SRFE layer, a third Conv convolutional layer, a fourth SRFE layer, and an SPPF+C2PSA layer. These layers are sequentially connected in series to form a linear network structure. The input image is fed into the Input layer. The first SRFE layer outputs a first-level feature map, the second SRFE layer outputs a second-level feature map, the third SRFE layer outputs a third-level feature map, and the SPPF+C2PSA layer outputs a fourth-level feature map. The size of the input layer is W×H×3, the size of the first SRFE layer is W / 4×H / 4×128, the size of the second SRFE layer is W / 8×H / 8×256, the size of the third SRFE layer is W / 16×H / 16×512, the size of the fourth SRFE layer is W / 32×H / 32×1024, and the size of the SPPF+C2PSA layer is W / 32×H / 32×1024.

[0044] The Stem layer is the initial downsampling layer, consisting of two consecutive convolutional layers, used to complete the initial feature extraction and resolution compression of the input image; the multiple SRFE layers are all feature extraction modules based on structure reparameterization (SRFE), which adopt a multi-branch convolutional structure during the training phase and are equivalently reparameterized to a single-path convolutional structure during the inference phase; the SPPF+C2PSA layer is a module in the YOLO11 network, consisting of an SPPF layer and a C2PSA layer, where the SPPF layer is a general pooling module and the C2PSA layer is an attention module.

[0045] The backbone network first processes the input image, then gradually obtains hierarchical feature maps of different scales through multi-level downsampling, and further expands the receptive field and aggregates contextual information in the deep stage. The SRFE layer is a structure-reparameterized feature extraction module. In the model training stage, the SRFE module adopts a multi-branch convolutional structure to enhance the joint modeling ability of fine-grained texture of needle marks, edge gray-level changes and deep semantic information. In the inference stage, the multi-branch convolutional structure in the SRFE module is equivalently reparameterized into a compact single-path convolutional structure, thereby reducing the inference computational overhead while maintaining the feature extraction accuracy.

[0046] The neck network includes a feature enhancement module and a spatial-channel reconstruction module. The feature enhancement module includes a first FEM feature enhancement layer, a second FEM feature enhancement layer, a third FEM feature enhancement layer, a fourth FEM feature enhancement layer, a first fusion layer, a second fusion layer, a third fusion layer, and a fourth fusion layer. The first, second, third, and fourth level feature maps are respectively input into the first, second, third, and fourth FEM feature enhancement layers for feature enhancement, resulting in a first feature map, a second feature map, a third feature map, and a fourth feature map, respectively. Specifically, the first level feature map is input into the first FEM feature enhancement layer for feature enhancement, resulting in a first feature map; the second level feature map is input into the second FEM feature enhancement layer for feature enhancement, resulting in a second feature map; the third level feature map is input into the third FEM feature enhancement layer for feature enhancement, resulting in a third feature map; and the fourth level feature map is input into the fourth FEM feature enhancement layer for feature enhancement, resulting in a fourth feature map.

[0047] The FEM feature enhancement layer is a feature enhancement layer composed of convolutional feature extraction and channel attention weighting. The input features of the FEM feature enhancement layer are first mapped to the number of output channels c2 through a convolution, and then batch normalized to obtain the main branch features. Then, using the main branch features as input, channel attention branches are formed by global average pooling, two layers of 1×1 convolution, ReLU activation function and Sigmoid activation function to generate channel weights of size (B, c2, 1, 1). Finally, the weights are multiplied with the main branch features channel by channel, and SiLU activation is used to obtain the enhanced output features, which is the feature map output by the FEM feature enhancement layer.

[0048] The third and fourth feature maps, after average pooling, are both input into the fourth fusion layer for fusion to obtain the fourth fused feature map. The second feature map, the upsampled fourth feature map, the upsampled fourth fused feature map, and the third feature map, after average pooling, are all input into the third fusion layer for fusion to obtain the third fused feature map. The first feature map, the upsampled third feature map, the upsampled third fused feature map, and the second feature map, after average pooling, are all input into the second fusion layer for fusion to obtain the second fused feature map. The upsampled second feature map, the upsampled second fused feature map, and the first feature map are all input into the first fusion layer for fusion to obtain the first fused feature map.

[0049] In the fusion stage of hierarchical feature maps at multiple scales, the Neck network introduces a FEM feature enhancement module as an FEM feature enhancement layer to perform cross-layer weighted fusion of hierarchical feature maps at different scales. The FEM feature enhancement layer adaptively adjusts the contribution ratios of high-level features, low-level features, and current-level features through learnable weights. High-level features provide richer semantic information and a wider contextual awareness range, low-level features preserve finer spatial textures and edge details, and current-level features maintain stable representations at the current scale. The weighted hierarchical feature maps are further fused through concatenation and convolution to achieve complementary fusion, resulting in multiple fused feature maps. This mitigates the problem of small needle mark features being easily submerged by large-scale background information in conventional equal-weighted fusion methods, improving the effective representation capability of dense, small targets.

[0050] The spatial-channel reconstruction module includes a first C3K2 layer, a second C3K2 layer, a third C3K2 layer, a fourth C3K2 layer, a first SCRU spatial-channel reconstruction unit, a second SCRU spatial-channel reconstruction unit, a third SCRU spatial-channel reconstruction unit, and a fourth SCRU spatial-channel reconstruction unit. The first, second, third, and fourth fused feature maps are respectively input into the first, second, third, and fourth C3K2 layers for convolution operations, yielding a first convolution output, a second convolution output, a third convolution output, and a fourth convolution output, respectively. The first convolution output and the upsampled second convolution output are both input into the first SCRU spatial-channel reconstruction unit for reconstruction, resulting in a first different... The first convolutional output after average pooling, the third convolutional output after upsampling, the first feature map at different scales after average pooling, and the second convolutional output are all input into the second SCRU spatial-channel reconstruction unit for reconstruction to obtain the second feature map at different scales. The second convolutional output after average pooling, the fourth convolutional output after upsampling, the second feature map at different scales after average pooling, and the third convolutional output are all input into the third SCRU spatial-channel reconstruction unit for reconstruction to obtain the third feature map at different scales. The third convolutional output after average pooling, the third feature map at different scales after average pooling, and the fourth convolutional output are all input into the fourth SCRU spatial-channel reconstruction unit for reconstruction to obtain the fourth feature map at different scales.

[0051] The C3K2 layer is a foundational layer in the YOLO11 network. It is a lightweight bottleneck aggregation layer based on the C2f / CSP approach, consisting of an input convolutional mapping, several internal bottleneck sub-blocks, and a concatenated output. It is used for local feature extraction and channel aggregation. The SCRU spatial-channel reconstruction unit consists of a spatial reconstruction sub-unit (SRU) and a channel reconstruction sub-unit (CRU). Input features are first input to the SRU, where group normalization evaluates the spatial features. A gating mechanism divides the features into information-preserving and redundancy-suppressing parts, and then cross-reconstructs to form spatial reconstruction features. Subsequently, the spatial reconstruction features are input to the CRU, which divides them along the channel dimension into two parts. One part is compressed using 1×1 convolution and then transformed using group convolution (GWC) and pointwise convolution (PWC). The other part is compressed using 1×1 convolution and then fused with the preserved channels. The two processed features are then fused to obtain fused features. Finally, the fused features are weighted using global pooling to complete channel reconstruction, outputting reconstructed feature maps at different scales.

[0052] In the background suppression and feature reconstruction stage, the Neck network introduces a SCRU spatial-channel reconstruction unit, which further inputs multiple fused feature maps into the SCRU spatial-channel reconstruction unit. The SCRU spatial-channel reconstruction unit first evaluates the importance of feature responses in the spatial dimension, enhancing high-information target regions and suppressing invalid background responses. Subsequently, it reorganizes and compresses the fused feature maps in the channel dimension, preserving the representational ability of high-value channels through grouped convolutions and compressing low-value or redundant channel responses through pointwise convolutions. Through this spatial dimension screening and channel dimension reconstruction process, the interference of complex chip backgrounds, noise textures, and spurious responses on the detection results can be reduced, and discrimination cues such as pin mark edges, contours, and grayscale differences under weak contrast conditions can be enhanced.

[0053] The detection output network head is composed of multiple detection heads of different scales optimized by fusing anchor frames. The detection output network head includes a first detection head, a second detection head, a third detection head, and a fourth detection head. The first detection head, the second detection head, the third detection head, and the fourth detection head are detection heads of different resolutions. The first feature map of different scales, the second feature map of different scales, the third feature map of different scales, and the fourth feature map of different scales are sequentially input into the first detection head, the second detection head, the third detection head, and the fourth detection head, respectively, and the multiple feature maps of different scales are detected to obtain the first detection head, the second detection head, the third detection head, and the fourth detection head.

[0054] In the detection output stage, the detection output network head inputs multiple features of different scales, enhanced by the FEM feature enhancement module and the SCRU spatial-channel reconstruction unit, into multiple detection heads of different resolutions to complete pin mark category judgment and bounding box regression. Considering that the actual size of the pin mark target is significantly smaller than that of conventional targets, the detection output network head further combines the pin mark scale distribution features to optimize the preset anchor box for smaller targets, thereby improving the initial matching accuracy between the anchor box and the extremely small pin mark target. Through multiple detection heads of different resolutions and the anchor box optimization mechanism, the positioning accuracy and detection stability of the chip pin mark basic detection network model for extremely small targets can be improved, and higher quality initial prediction results can be provided for the subsequent target domain sample splitting and feature alignment.

[0055] In step two, several teacher models from the joint teacher model group perform collaborative inference on unlabeled samples in the target domain to obtain the average detection confidence and cognitive dissonance of each pinhole target in the unlabeled samples of the target domain. The joint teacher model group is set to have K teacher models, which perform forward inference on the unlabeled samples of the target domain respectively, and construct a deduplicated global candidate set Ω through spatial fusion. For the i-th pinhole target in the global candidate set Ω, the prediction probabilities of the K teacher models in the joint teacher model group for that pinhole target are extracted. , where is the detection confidence of the K teacher models for the needle mark target; if a teacher model fails to detect the needle mark at the target location, its predicted probability is set to 0; using a joint teacher model group instead of a single teacher model can reduce the impact of single-teacher random bias on the evaluation results of the target domain samples. To measure the overall reliability of the joint teacher model group for the same needle mark target, the average detection confidence C for that needle mark target is calculated. i The calculation process is as follows:

[0056]

[0057] In the formula, Let be the confidence level of the k-th teacher model for the i-th needle mark target.

[0058] The average detection confidence of the pinprick target only reflects the absolute level of the predicted score of the joint teacher model group, but it cannot distinguish whether the low score is due to the target itself being difficult to identify or due to obvious conflicts between different teacher models. Therefore, an entropy-based information decomposition mechanism is introduced to define the expected value of the classification entropy of a single teacher model as the uncertainty of the prediction of a single teacher model, and the Shannon entropy of the average detection confidence of each pinprick target is calculated as the total uncertainty of the joint prediction of each pinprick target. The difference between the total uncertainty of the joint prediction of each pinprick target and the uncertainty of the prediction of a single teacher model is defined as the cognitive divergence degree U of that pinprick target. dis,i The calculation is as follows:

[0059]

[0060]

[0061]

[0062] In the formula, U dis,i H represents the cognitive divergence of the i-th needle mark target. cls,i For the uncertainty of a single teacher model prediction for the i-th needle mark target, H pre,i Let U be the total uncertainty in the joint prediction of the i-th pinhole target. The cognitive divergence degree U of the pinhole target. dis,i The value of reflects the consensus level of the joint teacher model group on the features of the current instance. The larger the value, the more obvious the prediction conflict between different teacher models on the current pin mark target, which usually corresponds to background noise, local texture or strong interference area. When the value remains low, it indicates that the joint teacher model group has reached a high consensus on the instance, and such samples are more suitable as high-quality pseudo-labels or cross-domain alignment objects.

[0063] Based on the average detection confidence and cognitive disagreement of each pin mark target, pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set S. reliable Drift sample set S drift and low-quality noisy sample set S noise The calculation process is as follows:

[0064]

[0065]

[0066]

[0067] In the formula, C high and C low U represents the high threshold and low threshold of the average detection confidence, respectively. high and U low S represents the high threshold and low threshold of cognitive disagreement, respectively; reliable For a highly reliable sample set, S drift For the drift sample set, S noise Let be a low-quality, noisy sample set, i be the i-th pin mark target in the unlabeled samples of the target domain, and Ω be the global candidate set in the unlabeled samples of the target domain. The high-reliability sample set can provide stable semantic supervision for the target domain; the drift sample set retains the distribution differences of the new production line in terms of illumination, texture, and noise, thereby avoiding the simple discarding of low-confidence but high-value samples and improving the utilization efficiency of unlabeled target domain data.

[0068] Obtain a highly reliable sample set S reliable Drift sample set S drift and low-quality noisy sample set S noise Then, the highly reliable sample set is used to generate pseudo-labels for supervised learning, constructing the target domain pseudo-label loss; the drift sample set is used as a carrier of the target domain's statistical distribution information to extract cross-domain features and multi-scale MMD and CMD feature alignment between the source and target domains, constructing the feature alignment loss; the target domain pseudo-label loss, feature alignment loss, and source domain supervised loss are weighted and fused to obtain the total loss; the low-quality noise sample set is not used in the current round of optimization. The target domain pseudo-label loss is constructed using the pseudo-labels generated from the highly reliable sample set as input and the standard supervised learning loss; the standard supervised learning loss can be the cross-entropy loss, and the calculation process can be expressed as follows:

[0069]

[0070] In the formula, S reliable A highly reliable sample set; For example, the cross-entropy loss for classification tasks or the mean squared error loss for regression tasks; f θ For the current student model, x t For the t-th sample in the highly reliable sample set, y t Generate the t-th pseudo-label for the highly reliable sample set.

[0071] Considering that cross-domain offsets in chip images manifest not only as mean-level shifts caused by illumination, exposure, and overall appearance style, but also as conditionally dependent structural changes caused by pad surface texture and high-frequency noise, a combined Maximum Mean Difference (MMD) and Conditional Mean Difference (CMD) approach is adopted, simultaneously compensating for both MMD and CMD offsets across multiple scale layers. Edge distribution alignment based on MMD is performed to maintain consistency with the multi-scale detection structure of the chip pin mark detection network model. Instance-level feature alignment is performed using feature maps at scales P3, P4, and P5, with the source domain instance-level feature set at the l-th scale layer defined as... The set of instance features of the target domain drift sample set is The maximum mean difference (MMD) loss at this scale is defined as follows:

[0072]

[0073] In the formula, It is represented as a mapping function to the reproducing kernel Hilbert space. Represented as the regenerating nucleus Hilbert space, and These represent the number of instance-level features in the source and target domains at the l-th scale layer, respectively. This term is mainly used to align the overall offset at the mean level, and is therefore more suitable for compensating for differences in brightness, contrast, and global appearance style. The feature maps at the three scales P3, P4, and P5 represent feature maps at three scales obtained after downsampling the input image by 8x, 16x, and 32x, respectively, which are feature maps with resolutions of W / 8×H / 8, W / 16×H / 16, and W / 32×H / 32.

[0074] Since Maximum Mean Difference (MMD) only aligns the edge distribution, it still cannot guarantee the consistency of needle mark feature distribution under the same semantic conditions. Therefore, Conditional Mean Difference (CMD) is introduced to constrain the conditional distribution. Based on the conditional distribution alignment of CMD, let the feature space be... The tag space is Then the conditional distribution can be mapped to the tensor product Hilbert space, and a conditional mean embedding operator for the source and target domains at scale l can be constructed based on this. and The conditional mean difference (CMD) loss at this scale is defined as follows:

[0075]

[0076] In the formula, The conditional mean difference (CMD) loss is denoted as the Hilbert-Schmidt norm. It is used to constrain the conditional means of instance-level features in the source and target domains to remain consistent in high-dimensional space under the same semantic conditions, thereby reducing cross-domain conditional distribution shift. Although the drift sample set is not directly used as high-quality pseudo-label supervision samples, its low joint cognitive divergence indicates that the joint teacher model group maintains a strong consensus on its structural semantics. Therefore, it can still serve as a source of approximate conditional information for constructing the aforementioned conditional distribution constraints. Specifically, based on the true labels of the source domain and the pseudo-labels of the target domain, instance-level features are divided into corresponding conditional subsets. Subsequently, a feature Gram matrix is ​​constructed on each conditional subset, and a regularization matrix is ​​introduced to obtain the empirical operator for conditional mean embedding. Finally, numerical calculation is completed through trace operations.

[0077] To simultaneously consider edge distribution shifts and conditional distribution shifts at different scales, the maximum mean difference (MMD) loss and the conditional mean difference (CMD) loss at the three scales are weighted and summed to obtain the feature alignment loss; the calculation process is shown below:

[0078]

[0079] In the formula, α and β represent the weight coefficients of the edge distribution alignment term and the conditional distribution alignment term, respectively. Considering the specific imaging characteristics of pin marks in chip images, the target domain offset often comes not only from changes in overall brightness, but also depends more on changes in local micro-texture and noise structures. Therefore, in the edge distribution alignment term and the conditional distribution alignment term, it is preferable to give the conditional distribution alignment term a higher weight. Thus, the high-reliability sample set provides stable supervision at the semantic level, and the drift sample set provides cross-domain correction signals at the feature level. Together, they constitute a dual-pathway mechanism for target domain adaptation.

[0080] The source domain supervised loss is the standard supervised learning loss of the model on the source domain data, and the calculation process is as follows:

[0081]

[0082] In the formula, D s For the source domain dataset, x s Let y be the s-th sample in the source domain dataset. s The true label is the s-th sample in the source domain dataset. For example, the cross-entropy loss for classification tasks or the mean squared error loss for regression tasks; f θ This is the current student model to be trained.

[0083] The target domain pseudo-label loss, feature alignment loss, and source domain supervision loss are weighted and fused to obtain the total loss. The initial student model is then trained again under supervised conditions, and the model parameters are updated by backpropagation using the total loss during the supervised training process, resulting in an updated student model. Simultaneously, the target domain pseudo-label loss, feature alignment loss, and source domain supervision loss are introduced to form a unified joint optimization objective, resulting in the total loss. The source domain supervision loss maintains the model's basic ability to distinguish needle marks, the target domain pseudo-label loss on a highly reliable sample set guides the model to learn stable discriminative features in the target domain, and the feature alignment loss on a drift sample set absorbs statistical offset information caused by changes in the target domain's operating conditions. The total loss L... total The calculation process is as follows:

[0084]

[0085] In the formula, L sup The source domain supervision loss is the detection loss on labeled samples in the source domain, used to maintain the model's basic ability to distinguish needle mark targets; L pseudo The target domain pseudo-label loss is the pseudo-label detection loss on highly reliable target domain samples, used to guide the model to learn stable semantic patterns in the target domain; L align The feature alignment loss is used to absorb the domain offset information contained in the drifted samples; λ pseudoλ represents the weighting coefficients of the pseudo-label loss in the target domain. align These are the weighting coefficients for the feature alignment loss. Finally, the initial student model is adjusted based on the total loss L. total Backpropagation is performed to update parameters, thereby gradually adapting to the new distribution in the target domain while maintaining the discriminative ability of the source domain.

[0086] In step four, after obtaining the updated student model, the model parameters of the joint teacher model group are iteratively updated using a differentiated EMA mechanism. Then, the updated joint teacher model group is used to perform collaborative inference on unlabeled samples in the target domain, dividing the pin marks in the unlabeled samples into a high-reliability sample set, a drift sample set, and a low-quality noise sample set. The student model is then updated again, and the updated student model is used to iteratively update the model parameters of the joint teacher model group using a differentiated EMA mechanism. This process is repeated until the maximum number of iterations is reached, resulting in the optimal student model. This allows the joint teacher model group to absorb stable knowledge from the student model through the differentiated EMA mechanism.

[0087] Let the parameters of the k-th teacher model at time t be... The parameters of the student model at time t+1 are: The iterative update of the k-th teacher model is as follows:

[0088]

[0089] In the formula, α k α is the momentum coefficient corresponding to the k-th teacher model, and different teacher models use different α. k The values ​​are chosen to create smooth states with varying speeds over time; K represents the number of teacher models in the joint teacher model group. Unlike single-momentum differential EMA, the differential EMA mechanism allows some teacher models to retain historically stable knowledge while others absorb the latest student model information, thereby improving the robustness of the joint teacher model group under different target domain conditions; when the final optimized student model parameters are obtained... and joint teacher model group parameters Afterwards, the system further enters the stage of outputting detection results and online self-evolving deployment. Among them, the differentiated EMA method is an improvement on the traditional exponential moving average EMA, which achieves differentiated updates of data or model parameters by dynamically adjusting the smoothing coefficients of different samples or features.

[0090] After multiple rounds of self-evolutionary training, an optimal student model is obtained. This optimized model is then used to perform pin mark detection on the target domain chip image, yielding the pin mark detection results. For each detected pin mark, the system output includes at least the coordinates of the top-left corner of the detection box, the box width, the box height, the target center coordinates, the detection confidence score, the image number, and the batch number. Based on industrial software interface requirements, the detection results can be further converted into standardized text records, structured tables, or database records. To facilitate subsequent re-inspection, model retraining, and production line traceability, pixel coordinates are preferably converted to a normalized center coordinate format.

[0091] ,

[0092] In the formula, x norm and y norm The coordinates represent the normalized center coordinates of the detection box, x and y represent the pixel coordinates of the top left corner of the pin mark detection box, w and h represent the width and height of the pin mark detection box, respectively, and W and H represent the width and height of the original chip image. Through the above standardized output method, the detection results can be directly used for subsequent incremental model training, defect statistical analysis, and industrial software interface integration.

[0093] To meet the requirements of continuous production line operation and stable model updates, this invention preferably employs a rolling update approach to deploy the self-evolving model. Specifically, the system retains checkpoints for the teacher and student models generated in each training round and performs rapid validation on a preset stable sample set. When the new model falls below a preset threshold in any of the following areas: average detection confidence, source domain benchmark detection performance index, or target domain stability index, the system reverts to the previous version of the model to avoid significant performance degradation caused by abnormal batch samples. Simultaneously, newly added unlabeled chip images are continuously added to the target domain sample pool to support the model's continuous self-evolution under different probe stations, chip types, and illumination conditions.

[0094] The above description is merely a preferred embodiment of the present invention. Those skilled in the art can make several modifications and optimizations based on the above disclosure without departing from the basic principles described above. These modifications and optimizations should be considered within the scope of protection as understood by the present invention.

Claims

1. A precise chip pin mark detection method based on semi-supervised self-evolution, characterized in that: Includes the following steps: Step 1: Introduce the FEM feature enhancement module and SCRU spatial-channel reconstruction unit to construct a basic chip pin mark detection network model. Train the basic chip pin mark detection network model with sufficient supervised pin mark samples in the source domain to obtain the initial student model. During the source domain training process, select snapshots of stable teacher models at different training stages and construct a joint teacher model group from the snapshots of stable teacher models at different training stages. The SCRU space-channel reconfiguration unit consists of a space reconfiguration subunit SRU and a channel reconfiguration subunit CRU. Step 2: Using several teacher models from the joint teacher model group, collaborative reasoning is performed on the unlabeled samples in the target domain to obtain the average detection confidence and cognitive dissonance of each pin mark target in the unlabeled samples of the target domain; based on the average detection confidence and cognitive dissonance of each pin mark target, the pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set, a drift sample set, and a low-quality noise sample set; The difference between the total uncertainty of the joint prediction of each pinprick target and the uncertainty of the prediction by a single teacher model is defined as the cognitive dissonance of that pinprick target. Step 3: Use the highly reliable sample set to generate pseudo-labels for supervised learning and construct the target domain pseudo-label loss; use the drift sample set for multi-scale MMD and CMD feature alignment between the source and target domains and construct the feature alignment loss; weight and fuse the target domain pseudo-label loss, feature alignment loss, and source domain supervised loss to obtain the total loss, and then conduct supervised training on the initial student model again. During the supervised training process, the initial student model performs backpropagation on the total loss to update the model parameters, resulting in an updated student model. The feature alignment loss is obtained by weighted summing of the maximum mean difference (MMD) loss and the conditional mean difference (CMD) loss across the three scales; based on the conditional distribution alignment using the conditional mean difference (CMD), the feature space is set as follows. The tag space is Then the conditional distribution can be mapped to the tensor product Hilbert space, and a conditional mean embedding operator for the source and target domains at scale l can be constructed based on this. and The conditional mean difference (CMD) loss at this scale is defined as follows: In the formula, Represented as the Hilbert-Schmidt norm; Step 4: After obtaining the updated student model, iteratively update the model parameters of the joint teacher model group using the differentiated EMA method until the maximum number of iterations is reached to obtain the optimal student model; use the optimal student model to perform pin mark detection on the target domain chip image to obtain the pin mark detection results of the target domain chip image.

2. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 1, characterized in that: The chip pin mark detection network model includes a backbone network, a neck network, and a detection output network. The preprocessed input image is input into the backbone network, which performs multi-level downsampling to gradually obtain multiple hierarchical feature maps at different scales. These hierarchical feature maps are then input into the neck network, where an FEM feature enhancement module and a SCRU spatial-channel reconstruction unit are introduced to fuse and enhance the hierarchical feature maps at different scales, resulting in multiple feature maps at different scales. Finally, these feature maps are input into multiple detection heads at different resolutions in the detection output network to obtain the pin mark detection results for the input image.

3. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 2, characterized in that: The backbone network includes an Input layer, a Stem layer, a first SRFE layer, a first Conv convolutional layer, a second SRFE layer, a second Conv convolutional layer, a third SRFE layer, a third Conv convolutional layer, a fourth SRFE layer, and an SPPF+C2PSA layer. These layers are sequentially connected in series to form a linear network structure. The input image is fed into the Input layer. The first SRFE layer outputs a first-level feature map, the second SRFE layer outputs a second-level feature map, the third SRFE layer outputs a third-level feature map, and the SPPF+C2PSA layer outputs a fourth-level feature map.

4. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 3, characterized in that: The neck network includes a feature enhancement module and a spatial-channel reconstruction module. The feature enhancement module includes a first FEM feature enhancement layer, a second FEM feature enhancement layer, a third FEM feature enhancement layer, a fourth FEM feature enhancement layer, a first fusion layer, a second fusion layer, a third fusion layer, and a fourth fusion layer. The first, second, third, and fourth layer feature maps are respectively input into the first, second, third, and fourth FEM feature enhancement layers for feature enhancement, resulting in the first, second, third, and fourth feature maps, respectively. The third feature map, after average pooling, is then processed... The first feature map and the fourth feature map are both input into the fourth fusion layer for fusion to obtain the fourth fused feature map. The second feature map after average pooling, the fourth feature map after upsampling, the fourth fused feature map after upsampling, and the third feature map are all input into the third fusion layer for fusion to obtain the third fused feature map. The first feature map after average pooling, the third feature map after upsampling, the third fused feature map after upsampling, and the second feature map are all input into the second fusion layer for fusion to obtain the second fused feature map. The second feature map after upsampling, the second fused feature map after upsampling, and the first feature map are all input into the first fusion layer for fusion to obtain the first fused feature map.

5. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 4, characterized in that: The spatial-channel reconstruction module includes a first C3K2 layer, a second C3K2 layer, a third C3K2 layer, a fourth C3K2 layer, a first SCRU spatial-channel reconstruction unit, a second SCRU spatial-channel reconstruction unit, a third SCRU spatial-channel reconstruction unit, and a fourth SCRU spatial-channel reconstruction unit. The first, second, third, and fourth fused feature maps are respectively input into the first, second, third, and fourth C3K2 layers for convolution operations, yielding a first convolution output, a second convolution output, a third convolution output, and a fourth convolution output, respectively. The first convolution output and the upsampled second convolution output are both input into the first SCRU spatial-channel reconstruction unit for reconstruction, resulting in a first different... The first convolutional output after average pooling, the third convolutional output after upsampling, the first feature map at different scales after average pooling, and the second convolutional output are all input into the second SCRU spatial-channel reconstruction unit for reconstruction to obtain the second feature map at different scales. The second convolutional output after average pooling, the fourth convolutional output after upsampling, the second feature map at different scales after average pooling, and the third convolutional output are all input into the third SCRU spatial-channel reconstruction unit for reconstruction to obtain the third feature map at different scales. The third convolutional output after average pooling, the third feature map at different scales after average pooling, and the fourth convolutional output are all input into the fourth SCRU spatial-channel reconstruction unit for reconstruction to obtain the fourth feature map at different scales.

6. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 1, characterized in that: In step two, several teacher models from the joint teacher model group perform collaborative inference on unlabeled samples in the target domain to obtain the average detection confidence and cognitive dissonance of each pinhole target in the unlabeled samples of the target domain. The joint teacher model group is set to have K teacher models. The joint teacher model group performs forward inference on the unlabeled samples in the target domain separately and constructs a deduplicated global candidate set Ω through spatial fusion. For the i-th pinhole target in the global candidate set Ω, the prediction probability of the pinhole target by the K teacher models in the joint teacher model group is extracted as the detection confidence of the K teacher models for the pinhole target, and the average detection confidence C of the pinhole target is calculated. i The calculation process is as follows: In the formula, Let be the confidence level of the k-th teacher model for the i-th needle mark target.

7. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 6, characterized in that: An entropy-based information decomposition mechanism is introduced, defining the expected value of the classification entropy of a single teacher model as the uncertainty of the single teacher model's prediction, and calculating the Shannon entropy of the average detection confidence of each pinprick target as the total uncertainty of the joint prediction for each pinprick target; the difference between the total uncertainty of the joint prediction for each pinprick target and the uncertainty of the single teacher model's prediction is defined as the cognitive divergence degree U of that pinprick target. dis,i The calculation is as follows: In the formula, U dis,i H represents the cognitive divergence of the i-th needle mark target. cls,i For the uncertainty of a single teacher model prediction for the i-th needle mark target, H pre,i The total uncertainty in the joint prediction of the i-th needle mark target.

8. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 6, characterized in that: Based on the average detection confidence and cognitive disagreement of each pin mark target, pin mark targets in the unlabeled samples of the target domain are divided into a high-reliability sample set S. reliable Drift sample set S drift and low-quality noisy sample set S noise The calculation process is as follows: In the formula, C high and C low U represents the high threshold and low threshold of the average detection confidence, respectively. high and U low S represents the high threshold and low threshold of cognitive disagreement, respectively; reliable For a highly reliable sample set, S drift For the drift sample set, S noise Ω represents the set of low-quality, noisy samples, and Ω represents the global candidate set among the unlabeled samples in the target domain.

9. The method for accurate detection of chip pin marks based on semi-supervised self-evolution as described in claim 6, characterized in that: In step four, after obtaining the updated student model, the model parameters of the joint teacher model group are iteratively updated using the differentiated EMA method until the maximum number of iterations is reached, and the optimal student model is obtained. Let the parameters of the k-th teacher model at time t be... The parameters of the student model at time t+1 are: The iterative update of the k-th teacher model is as follows: In the formula, α k Let be the momentum coefficient corresponding to the k-th teacher model, and K be the number of teacher models in the joint teacher model group.

Images