High-precision and high-adaptability intelligent detection and diagnosis method and system for electronic product defects

Abstract

The invention relates to the technical field of detection and diagnosis of surface and subsurface defect faults of electric power products, and discloses a high-precision and high-adaptability intelligent detection and diagnosis method and system for electronic product defects, and the method comprises the steps: obtaining image data of chip packaging and electrical equipment, and building a training data set according to the image data; fault detection models suitable for chip packaging and electrical equipment are constructed respectively; and training the fault detection model by using the training data set, constructing an evaluation index and calculating a loss function, and adjusting the fault detection model according to the evaluation index and the loss function. By implementing the method, the model robustness and the small target detection capability are enhanced, and the small target feature extraction capability is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of detection and diagnosis of surface and subsurface defects of power products, and in particular to a high-precision and high-adaptability intelligent detection and diagnosis method and system for electronic product defects. Background Art

[0002] PCBs are critical structures for electrical and pneumatic interconnection, signal transmission, mechanical connection, and support of power and electronic systems. They are also a major area of ​​component failure, especially in high-frequency and high-voltage circuits. Due to their complex multilayer structure, PCBs are susceptible to various defects during processing and use, such as layer separation, delamination, breakdown damage, and microvoids. Solder joints, in turn, constitute a critical component on PCBs. In addition to serving as electrical conduits, they also provide the mechanical connection between electronic components and the substrate. Solder joints are more susceptible to defects such as cracks, voids, and missing balls. These defects can adversely affect the performance and lifespan of flip-chip packages, leading to unstable and intermittent circuit behavior. This poses a significant risk to circuit commissioning, operation, and maintenance. Therefore, evaluating solder joint integrity is crucial for fault detection and maintenance in PCBs.

[0003] Traditional PCB defect detection technologies include manual visual inspection and automated optical inspection, such as X-rays, CT imaging, ultrasound, laser ultrasound, terahertz imaging, and thermal infrared imaging. In the detection of weak infrared targets, infrared images are susceptible to various issues, including thermal noise and scattering noise, due to thermal obstruction during processing and discontinuities in multiple frames. This noise and interference not only degrade image quality but can also obscure the signal of faint targets. Weak targets have extremely low signal-to-noise ratios, making their texture and structural shape particularly blurred in infrared images. Due to the weak target signal and susceptibility to noise interference, detection is relatively difficult. While manual visual inspection and automated optical inspection are the most common methods, they are limited to detecting visible surface defects and cannot guarantee the absence of internal defects. PCB defects inherently present challenges such as diverse circuit designs, small defect areas, and similar characteristics across defects. This complicates defect detection and makes false detections and missed detections more likely. Even current deep learning methods have not effectively addressed these issues.

[0004] With the growing trend toward thinner semiconductor wafers, various issues have emerged during the semiconductor silicon wafer production process, including significant issues related to surface and subsurface microcracks, ranging in size from a few microns to tens of microns. Semiconductor chip materials are inherently brittle and therefore susceptible to stress-induced cracking during chip manufacturing and assembly. These cracks primarily manifest as scratches, fractures, orange peel effects, and pits. For example, defects and failures in micron-scale multilayer chips include missed solder balls, cold solder joints, surface contaminants, internal contamination, packaging material aging, solder composition variations, electrical shorts, resistance anomalies, package looseness, component displacement, solder ball anomalies in critical areas, defects in different regions, placement offset, solder shorts, latent cracks, latent cold solder joints, and other defects. These defects, combined with material and structural failures, can adversely affect the performance and reliability of finished power electronic equipment, ultimately impacting the quality of silicon-based microelectronic products. Ensuring product quality and performance requires non-destructive testing of silicon wafers. Surface-mount components use solder bumps to interconnect the chip / package to the substrate or PCB. However, common manufacturing defects, including opens, cracks, or missing solder bumps, still exist. Since solder bumps are hidden within the package after assembly, and the trend toward higher density and ultra-fine pitch is increasing, existing detection technologies and image intelligence deep learning processing methods are not adaptable to these subtle surface and near-surface defects, making defect detection increasingly difficult and challenging. Detecting defects in solder bumps has become a critical issue in integrated circuit manufacturing. Faced with the rapidly evolving demands for the integration of computing power and electricity, internal and external faults are a frequent occurrence during solder assembly, inspection, and use of integrated circuits. Symptoms include: complete or partial functional failure: The integrated circuit may fail completely, or some functional modules may fail, resulting in device startup failure or malfunction. Performance degradation: For example, a severe loss of gain may cause the integrated circuit to lose its amplification capability. Signal interference: A damaged integrated circuit may generate noise or interference, affecting signal quality and stability, resulting in a reduced signal-to-noise ratio. Slowed response: Damaged integrated circuits can slow processing speeds, impacting device response times. Abnormal power consumption: This may cause an abnormal increase in power consumption, impacting the device's power management. Overheating: Integrated circuits may generate abnormal heat, causing the device to overheat and even pose a safety hazard. Appearance damage: Integrated circuits may show physical damage, such as cracks or burns. IC pins may also break or become poorly soldered, resulting in poor circuit connections and affecting normal device operation.

[0005] While some deep learning-based target detection methods have achieved remarkable results in image processing for visible light images, their performance is often hampered by data scarcity and feature differences in infrared images. The presence of heavy clouds and mist, blurred nighttime thermal infrared images, and the inherent characteristics of existing infrared imaging methods result in small, dark targets appearing in images, resulting in low signal-to-noise ratio, contrast, and brightness, making target detection more difficult. Furthermore, because small targets occupy fewer pixels in an image, their feature information is relatively weak, potentially leading to reduced detection accuracy and severely impacting the detection and location of faults in smart grids. Summary of the Invention

[0006] In view of this, the present invention provides a high-precision and high-adaptability intelligent detection and diagnosis method and system for electronic product defects, so as to solve the problem of difficulty and low accuracy in fault detection of power product defects.

[0007] In a first aspect, the present invention provides a high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects, the method comprising:

[0008] Acquire image data of chip packages and electrical devices, and establish a training data set based on the image data;

[0009] Fault detection models suitable for chip packaging and electrical equipment are constructed respectively. In the input module of the fault detection model, the input size of the fault detection model is determined according to the image size, and the mosaic data enhancement algorithm is used to optimize the small target detection capability. The SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability. Multi-scale feature fusion optimization and spatial pyramid pooling adjustment operations are performed in the neck module of the fault detection model. The attention mechanism of the SW-MSA module is adjusted in the prediction module of the fault detection model.

[0010] Training the fault detection model using the training data set, constructing an evaluation index and calculating a loss function, and adjusting the fault detection model according to the evaluation index and the loss function;

[0011] The fault detection model is used to perform defect fault detection and diagnosis on power products.

[0012] The present invention provides a high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects. It determines the appropriate input size according to the image size of the imaging device and adopts the Mosaic data enhancement algorithm to highlight the features of small targets and enrich the data set, thereby enhancing the robustness of the model and the ability to detect small targets. The convolution kernel size and stride are flexibly adjusted, including dynamic settings based on the feature map in the initial and subsequent convolution layers. The SW-MSA module is also introduced to effectively improve the ability to extract small target features. Multi-scale feature fusion optimization, adding smaller feature maps rich in detail information and adopting a weighted fusion strategy, while adding new operations to the pooling operation and adjusting the weights, etc., to ensure that the features of small targets are effectively represented in the channel dimension. In the SW-MSA module, the window size and the number of heads of the multi-head mechanism are adjusted, and the linear layer parameters are optimized to better reflect the global and local relationships of small targets.

[0013] In an optional embodiment, acquiring image data of a chip package and an electrical device and establishing a training data set based on the image data includes:

[0014] Acquire image data of chip packaging and electrical equipment, and perform histogram equalization processing on the image data;

[0015] Use image annotation tools to annotate image data and establish corresponding annotation types;

[0016] The labeled image data is used as training data to establish a training dataset.

[0017] In an optional embodiment, in the input module of the fault detection model, determining the input size of the fault detection model according to the image size and using the Mosaic data enhancement algorithm to optimize the small target detection capability include:

[0018] In the input module, the input size of the fault detection model is determined according to the image size;

[0019] The image selection model based on probability distribution assigns different selection probabilities to each image according to its category label and feature importance, and selects the corresponding category images for splicing in the Mosaic process according to the probability;

[0020] Based on the affine transformation matrix, the image's rotation angle, scaling ratio, translation position, and center of gravity are automatically determined according to its content features during stitching, so that it can be rotated by a preset angle and translated to a preset position in the Mosaic image.

[0021] For each adjacent image pair among multiple images participating in Mosaic stitching, the weights of the pixels in their boundary areas are calculated and the pixel values ​​are fused.

[0022] In an optional embodiment, a SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability, including:

[0023] Dynamically adjust the window size according to the size of the input feature map;

[0024] The number of windows is dynamically adjusted along the height and width directions according to the step size parameter.

[0025] In an optional embodiment, performing multi-scale feature fusion optimization and spatial pyramid pooling adjustment operations on the neck module of the fault detection model includes:

[0026] Increase the number of upsampling and downsampling, and set the size of multiple feature map fusion;

[0027] Adopt multiple aggregation strategies and assign different weights according to the importance of feature maps of different scales for small target detection;

[0028] In the spatial pyramid pooling part, set pooling operations of multiple spatial sizes;

[0029] In the stitching operation after pooling, the mapping method of the number of channels is adjusted according to the feature richness and target number of the image.

[0030] In an optional embodiment, adjusting the attention mechanism of the SW-MSA module in the prediction module of the fault detection model includes:

[0031] Calculate the target based on the self-attention within the window and adjust the window size;

[0032] Adjust the number of heads in a multi-head mechanism based on the target feature distribution of chip packages and electrical devices.

[0033] In an optional embodiment, the fault detection model is trained using the training data set, an evaluation index is constructed and a loss function is calculated, and the fault detection model is adjusted according to the evaluation index and the loss function, including:

[0034] During the training of the chip package fault detection model, the training dataset is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the model performance indicators on the validation set under different batch values.

[0035] Gradually increase the number of training rounds based on the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance indicators meet the preset conditions;

[0036] Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate;

[0037] Adopt a learning rate decay strategy and adjust the decay rate and period according to the performance changes of the model on the validation set;

[0038] Performing a linear transformation on the feature vectors within the window to obtain a query vector, a key vector, and a value vector using an adaptively learned weight matrix, wherein the weight matrix is ​​adaptively adjusted during training according to the back propagation of the loss function;

[0039] For each head, a learnable bias matrix is ​​introduced to perform multi-head attention calculation in each window to obtain the attention score;

[0040] Perform weighted summation of the attention score and value vector to obtain the single-head output feature vector;

[0041] The output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

[0042] In an optional embodiment, the fault detection model is trained using the training data set, an evaluation index is constructed and a loss function is calculated, and the fault detection model is adjusted according to the evaluation index and the loss function, further comprising:

[0043] In the process of training the fault detection model for electrical equipment, the training data set is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the performance indicators of the model on the validation set under different batch values.

[0044] Gradually increase the number of training rounds based on the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance indicators meet the preset conditions;

[0045] Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate;

[0046] A small target detection penalty term is added to CIOU_LOSS. The weight coefficient is determined according to the ratio of target size to image size, which increases the weight of small target position and category error. At the same time, Focal Loss is used to adjust the adjustable parameters to solve the category imbalance problem.

[0047] Introducing multiple evaluation indicators to assess the performance of the model in flip-chip electrical equipment detection in order to adjust and optimize the model;

[0048] For each head, a learnable bias matrix is ​​introduced to perform multi-head attention calculation in each window to obtain the attention score;

[0049] Perform weighted summation of the attention score and value vector to obtain the single-head output feature vector;

[0050] The output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

[0051] In a second aspect, the present invention provides a high-precision and high-adaptability intelligent detection and diagnosis system for electronic product defects, the system comprising: an active excitation infrared imaging zoom camera, an industrial-grade CMOS sensor, an excitation unit, a laser driver, a CW laser, a reflector, a programmable logic controller, a cylindrical lens, a wire beam exciter, a thermal propagation device,

[0052] An active-excitation infrared imaging zoom camera captures thermal images of chip packaging and electrical equipment, and divides the thermal images into a training learning image group and a test detection image group in proportion. Images of defective faults and non-defective faults in the two groups are added to the YOLOV5 algorithm system respectively. The thermal images are classified by product, and then the classified images are used to implement the training learning set and detection test set of the YOLOV5 algorithm respectively. After the quality of the training learning set and the detection image meets the performance evaluation indicators, the images are inspected to identify the fault defect location and classification.

[0053] The present invention provides a high-precision and high-adaptability intelligent detection and diagnosis system for electronic product defects. It adopts an active infrared thermal imaging system and a YOLOV5-CDF-Mosaic-SW-MSA image depth processing method to perform intelligent non-destructive detection and diagnosis of surface and sub-surface defects in power and electronic system products. It detects, diagnoses and visually tracks surface and sub-surface defects through deep learning.

[0054] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to the first aspect or any corresponding embodiment thereof.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 1 is a flow chart of a method for intelligent detection and diagnosis of defects in electronic products with high precision and high adaptability according to an embodiment of the present invention;

[0058] Figure 2 This is an infrared image of an integrated passive NiCr resistor on a flip-chip substrate;

[0059] Figure 3 It is a schematic diagram of the test results of multi-layer chips;

[0060] Figure 4 The infrared images show a chip defect with an accuracy of 99.8% and a crystal defect with an accuracy of 99.9% on the PCB board.

[0061] Figure 5 The tracking chip on the power transformer control PCB board has an accuracy of 99.5% within the normal working temperature range, and an abnormal fault temperature of 75°C after 5 minutes of operation.

[0062] Figure 6 is the area ratio of the precision-recall curve;

[0063] Figure 7 1. A schematic diagram of a device for detecting and diagnosing defects and faults in power products and its workflow according to an embodiment of the present invention;

[0064] Figure 8 It is a visual analysis diagram of PCB application of the present invention;

[0065] Figure 9 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0067] In the related technologies, the red-hot external detection system and image processing algorithms R-CNN, YOLO algorithm, SSD algorithm, and Transformer intelligent algorithm are adapted to the detection of visually obvious defects under specific conditions, and are time-consuming and slow to detect, making them unsuitable for real-time detection tasks. Although these methods attempt to improve the quality of infrared images through image enhancement or filtering technology, they often cannot simultaneously solve problems such as low SNR, poor contrast, and noise interference. In addition, the strided convolution or pooling layer of the existing intelligent network will lead to the loss of fine information and insufficient features with poor learning effects. This enhancement function enables the network to more effectively extract and retain feature information related to microscopic defects. The mutual interference of various complex factors has led to the low performance evaluation indicators of the image and existing graphics processing of the red-hot external detection system (precision, recall rate, accuracy, and F1 score, etc.), and the high rate of false detection and missed detection. In particular, it is impossible to detect small defects (hidden) on the surface and sub-surface of the product and the existing power electronic systems. Online positioning detection and diagnosis of faults and defects in products cannot be achieved, which has become an industry problem.

[0068] According to an embodiment of the present invention, an embodiment of a high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] Before describing the steps, SW-MSA is explained; in Swin-Transformer, SW-MSA is the abbreviation of ShiftedWindow Multi-head Self-Attention, which means shifted window multi-head self-attention mechanism.

[0070] Multi-head self-attention: Traditional multi-head self-attention mechanisms map input features to multiple subspaces and compute attention in parallel, capturing relationships between different features and improving the model's expressiveness. In the image domain, this can be understood as capturing the relationships between image features from multiple perspectives.

[0071] Shifting windows: The Swin-Transformer divides the image into multiple non-overlapping local windows. Conventional windowed multi-head self-attention (W-MSA) only calculates attention within each window, lacking inter-window connections. SW-MSA shifts windows within adjacent Transformer blocks (typically by half the window size), allowing previously non-adjacent windows to interact with each other, expanding the receptive field and enhancing the model's ability to capture contextual information. Compared to global self-attention, SW-MSA significantly reduces computational effort.

[0072] Through SW-MSA, Swin-Transformer can maintain low computational complexity when processing large-size images, and can also effectively capture contextual information between different windows, performing well in computer vision tasks such as image classification, object detection, and semantic segmentation.

[0073] This embodiment provides a high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects. Figure 1 FIG. 1 is a flow chart of a method for intelligent detection and diagnosis of defects in electronic products with high precision and high adaptability according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0074] Step S1: Acquire image data of chip packages and electrical devices, and establish a training data set based on the image data.

[0075] Specifically, specialized high-precision optical microscopes and other imaging equipment are used to acquire image data of flip-chip packages and electrical devices. These images should include samples of various undesirable conditions (such as missing protrusions, cracks, voids, missing balls, and poor solder joints) as well as normal conditions to form a comprehensive training dataset.

[0076] In step S2, fault detection models suitable for chip packaging and electrical equipment are constructed respectively. In the input module of the fault detection model, the input size of the fault detection model is determined according to the image size, and the Mosaic data enhancement algorithm is used to optimize the small target detection capability. The SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability. Multi-scale feature fusion optimization and spatial pyramid pooling adjustment operations are performed in the neck module of the fault detection model. The attention mechanism of the SW-MSA module is adjusted in the prediction module of the fault detection model.

[0077] Specifically, the convolution kernel size and stride are adjusted in the basic module (Backbone): in the initial convolution layer (similar to the Conv_6x6 module in the original solution), the convolution kernel size is set to 3x3 or 4x4, and the stride is set to a value between 0.5 and 1. In subsequent convolution layers, the convolution kernel size and stride are dynamically adjusted according to the feature map size and target scale. For small feature maps close to the target size, a convolution kernel of 3x3 and a stride of 1 are used. The SW-MSA module is introduced in the Backbone module to enhance the model's ability to extract small target features. The neck module (Neck) performs multi-scale feature fusion optimization: smaller feature maps rich in detail information such as 10x10x2048 are added and fused with the original 20x20x1024, 40x40x512, and 80x80x256 feature maps. The fusion order and method are determined through experiments, and a weighted fusion strategy is adopted to give higher weights to small-sized feature maps and appropriately smaller weights to large-sized feature maps according to their contribution to the target context information.

[0078] Step S3: Use the training data set to train the fault detection model, construct evaluation indicators and calculate the loss function, and adjust the fault detection model according to the evaluation indicators and the loss function.

[0079] Specifically, during model training, a smaller learning rate is used, starting from 0.0001-0.001, and a learning rate decay strategy is adopted, gradually reducing the learning rate as the number of training rounds increases to ensure that the model can stably converge to a better state. The batch size is reduced to 8-16, so that the model can more fully utilize the feature information of each sample when updating parameters each time, avoiding the averaging of small target features due to excessively large batches. Based on the model convergence and performance improvement trend, the number of training rounds is appropriately increased by 20-50% from the original plan to ensure that the model can fully learn the complex features of small targets and achieve better detection results. A small target detection penalty term is added to CIOU_LOSS, and the weight coefficient is determined based on the ratio of target size to image size. The weight of small target position and category error is increased. At the same time, Focal Loss is used to adjust parameters to address the problem of class imbalance.

[0080] Based on the CIOU_LOSS used in the original solution, a penalty term for small object detection is added. For example, the position error and category error of detected small objects (such as solder balls and bumps) are given higher weights to emphasize the accuracy of small object detection. At the same time, FocalLoss can be used to address the class imbalance problem that may exist in flip-chip package inspection, especially for rare adverse conditions (such as voids), so that the model pays more attention to difficult-to-detect categories.

[0081] In addition to commonly used metrics such as accuracy, recall, and mAP-50, we introduce metrics specifically tailored to small object detection and high-precision detection, such as mean intersection over union (mIOU), small-object recall, and area under the precision-recall curve (AUPRC). These metrics allow for a comprehensive evaluation of the model's performance in flip-chip package inspection, enabling better model tuning and optimization.

[0082] The optimized loss function (adding a small target detection penalty term and using Focal Loss) and the introduced new evaluation metrics can better handle category imbalance, make the model more focused on rare adverse situations, comprehensively and accurately evaluate model performance, and solve the category imbalance problem.

[0083] Step S4: Utilize the fault detection model to perform defect fault detection and diagnosis on the power product.

[0084] Specifically, during the detection phase, the flip-chip package image to be inspected is input into the trained model to identify defects such as missing protrusions, cracks, voids, missing balls, and cold solder joints in the image, and accurately output the category of each defect and its corresponding bounding box coordinates and other information.

[0085] Pay attention to rare adverse events

[0086] Recall (R): This represents the ratio of correctly predicted detection boxes to actual ground-truth boxes among all the detection boxes predicted by the model. It reflects the model's ability to detect real objects (avoiding missed detections).

[0087]

[0088] TP-(True Positive) is the number of correctly detected targets. FN-(False Negative) is the number of missed targets.

[0089] Precision (P): This represents the proportion of correctly predicted detection boxes (positive samples) among all the detection boxes predicted by the model. It reflects the reliability of the model's predictions of positive samples (avoiding false positives).

[0090]

[0091] FP (False Positive), the number of targets misdetected by the model (detection boxes that are "created out of thin air" or "wrongly matched").

[0092] Average Precision (mAP-50): AP measures the effectiveness of detection for a specific category. Different confidence levels and IoU thresholds correspond to different precision and recall rates. The area of ​​the two-dimensional curve formed by these two rates is the AP value. The average of the AP values ​​across different categories is mAP, which measures the effectiveness of detection across multiple categories.

[0093]

[0094] Introducing Small-Object-Recall

[0095] The number of true positives in a small target is TP SmaII , the actual number of positive examples of small targets is TP small +FN Smail , then the small target recall rate (SOR) formula is:

[0096]

[0097] Mean cross-well ratio (mloU)

[0098] For a single target bounding box prediction result and a real box, let their intersection area be I and their union area be U, then the intersection and union ratio is A series of IoU values ​​are calculated for the bounding boxes of all small targets. Assuming there are m small targets, the mlOU calculation formula is:

[0099] Area under the precision-recall curve (AUPRC)

[0100] Similar to calculating AP, but here there are no restrictions on conditions such as IoU threshold. The precision-recall rate curve (PR curve) is directly drawn according to the accuracy values ​​corresponding to the model prediction results at different recall rates, and then the area under the curve is calculated by numerical calculation methods such as integration to obtain AUPRC.

[0101] The present invention provides a high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects. It determines the appropriate input size according to the image size of the imaging device and adopts the Mosaic data enhancement algorithm to highlight the features of small targets and enrich the data set, thereby enhancing the robustness of the model and the ability to detect small targets. The convolution kernel size and stride are flexibly adjusted, including dynamic settings based on the feature map in the initial and subsequent convolution layers. The SW-MSA module is also introduced to effectively improve the ability to extract small target features. Multi-scale feature fusion optimization, adding smaller feature maps rich in detail information and adopting a weighted fusion strategy, while adding new operations to the pooling operation and adjusting the weights, etc., to ensure that the features of small targets are effectively represented in the channel dimension. In the SW-MSA module, the window size and the number of heads of the multi-head mechanism are adjusted, and the linear layer parameters are optimized to better reflect the global and local relationships of small targets.

[0102] In an optional embodiment, step S1 includes:

[0103] Step S11 : acquiring image data of the chip package and the electrical device, and performing histogram equalization processing on the image data.

[0104] Step S12: annotate the image data using an image annotation tool and establish a corresponding annotation type.

[0105] Step S13: Use the labeled image data as training data to establish a training data set.

[0106] Specifically, a specialized optical microscope or other high-precision imaging device is used to acquire flip-chip package images. The images should contain various possible states, including normal state, missing protrusions, cracks, voids, missing balls, cold solder joints, surface contaminants, internal contamination, aging of packaging materials, changes in solder composition, electrical short circuits, abnormal resistance, loose packaging, component displacement, abnormal solder balls in key areas, defects in different areas (such as defects in the upper left corner area, defects in the center area), mounting offset, solder short circuits, potential cracks, potential cold solder joints, etc. The collected color or grayscale image is input and first converted into a grayscale image. The histogram of the original grayscale image is statistically calculated, the frequency of each pixel value is calculated, and then the cumulative distribution function (CDF) is calculated. According to the CDF, the original pixel values ​​are mapped to new pixel values ​​so that the histogram is approximately evenly distributed, thereby enhancing the image contrast. Use the Labelimg tool for annotation. The annotation types include: lq (missing solder ball), xh (cold solder joint), mqxs (missing protrusion), qtxb (deformed protrusion), lk (crack), kd (void), bmwr (surface contamination), nbwr (internal contamination), fbcl (package material aging), hldf (solder composition change), dqdl (electrical short), dzcy (resistance anomaly), fbss (package looseness), bjwy (component displacement), gjqy (critical area solder ball anomaly), zjsq (upper left corner area defect), zxqy (center area defect), tzpy (mounting offset), hjdl (solder short), qzlk (potential crack), and qxh (potential cold solder joint). Accurately mark the location and range of each anomaly. Position information can be described using a coordinate system or relative position. The severity of the anomaly should also be considered.

[0107] In addition to methods such as histogram equalization, a suitable combination of data enhancement methods can be selected based on the characteristics of the flip chip package image and the detection requirements. For example, small-angle rotation (such as -15° to 15°), small-scale scaling (such as 0.8-1.2 times), appropriate brightness adjustment, and adding a small amount of Gaussian noise can be used to increase data diversity while avoiding excessive changes in the key features of small targets, so that the model can better learn the characteristics of flip chip packages in different situations. Comprehensively collected and carefully labeled data allows the model to more accurately learn the characteristics of various flip chip package anomalies. Histogram equalization preprocessing helps improve image recognition and lays the foundation for subsequent accurate detection.

[0108] Classification and annotation based on physical structure and defect type:

[0109] Solder ball annotations. Missing Solder Ball (lq): This annotation is used when a solder ball location is clearly missing in the chip package image. Cold Solder (xh): If the connection between the solder ball and the chip or substrate appears weak, has gaps, or does not conform to normal solder joint configuration, it is annotated as a cold solder joint. This annotation allows the model to focus on the characteristics of the solder ball connection, which is critical for detecting solder quality issues. Bump Annotations. Missing Bump (mqxs): This annotation is used when a bump in a flip-chip package is missing. The integrity of the bump can have a significant impact on the function and stability of the chip. By learning this annotation type, the model can better determine whether the bump is normal. Deformed Bump (qtxb): This annotation is used when the shape of a bump is significantly different from its normal shape, such as bending or twisting. Crack Annotation (lk): Cracks are uniformly annotated, regardless of whether they occur on the chip surface, solder balls, or bumps. Void Annotation (kd): Voids are annotated when they are found within the chip package or between the solder ball and substrate.

[0110] Contamination-related status. Surface contamination (bmwr): If there are contaminants such as dust, oil, and chemical residue on the chip package surface, they need to be marked. These contaminants may affect the chip's heat dissipation and electrical performance, or cause corrosion and other problems. Further distinctions can be made during marking. Internal contamination (nbwr): Contamination within the chip package, such as impurities mixed into the packaging material, also needs to be marked.

[0111] Material property changes. Package material aging (FBCL): Chip packaging materials may age over time or under specific environmental conditions. Solder composition change (HLDF): Changes in the solder composition of the solder balls can affect soldering quality and electrical performance. For example, changes in the solder alloy composition due to oxidation or reaction with other substances should be noted, along with the location of such changes.

[0112] Electrical performance-related states. Electrical short circuit (dqdl): In addition to welding short circuits, electrical short circuits may occur between circuits or components inside the chip package due to various reasons (such as packaging defects, impurity conductivity, etc.). Marking the location of the electrical short circuit and the components that may be involved can help the model identify this fault state that seriously affects the chip function. Resistance abnormality (dzcy): When the resistance value of the solder ball, circuit or internal component of the chip changes abnormally, it is marked as resistance abnormality. This state may be caused by poor welding, material changes or component damage. The model can monitor the changes in the electrical performance of the chip by learning the marking of resistance abnormalities.

[0113] Mechanical stability status. Package looseness (fbss): If the connection between the chip package and the chip or substrate is loose, it is marked as package loose. This situation may cause the chip to be displaced or damaged when it is subjected to mechanical vibration or external force during use. Accurate marking allows the model to learn the characteristics of looseness and is used to detect the mechanical stability problems of the chip. Component displacement (bjwy): The position of components inside the chip package (such as chips, solder balls, passive components, etc.) is marked. Component displacement may affect the normal operation of the chip. By identifying this state, the model can evaluate the assembly quality and mechanical integrity of the chip.

[0114] Combine functional impact and location information annotation. Functional critical area annotation: For areas in the chip package that play a critical role in the function, such as the pin area where the chip is connected to the outside or the solder ball area related to power transmission, special annotations can be set. For example, "critical area solder ball abnormality (gjqy)" is used when the solder balls in these critical areas are missing or have cold solder joints. This allows the model to pay more attention to the status of these important areas, because the bad conditions in these areas may have serious consequences for the overall function of the chip. Position-related annotation: In addition to defect type annotation, it can also be combined with location information for annotation. For example, the chip package is divided into different quadrants or areas, and labeled as "upper left corner area defect (zjsq)", "center area defect (zxqy)", etc. This annotation method can help the model learn the defect distribution rules of different positions, and can locate the defect position more accurately during detection. It is especially suitable for situations where the defect position is related to the detection strategy.

[0115] Consider the production process and potential problem annotation. Process step related annotation: According to the production process steps of flip chip packaging, problems that may occur in each step are annotated. For example, after the chip mounting step, there may be a "mounting offset (tzpy)", that is, the chip position is offset compared to the normal position; after the welding step, there may be a "welding short circuit (hjdl)" problem. Annotating these process-related problems can make the model better adapt to the quality inspection needs in the production process. Potential problem prediction annotation: Some potential defects that may gradually develop into serious problems during the use of the chip can also be annotated. For example, if there are signs of tiny cracks or the solder ball connection is slightly loose, it is marked as "potential cracks (qzlk)" or "potential cold soldering (qxh)". In this way, these potential problems can be discovered in advance through the model, which helps to intervene before the chip suffers a serious failure.

[0116] Specialized high-precision imaging equipment is used to capture images of various possible flip-chip package anomalies, covering a wide range of conditions, from physical defects to electrical performance anomalies. The resulting data is rich and comprehensive. A specific histogram equalization operation is used to convert the captured images into grayscale and enhance contrast, facilitating subsequent identification of image features. The Labelimg tool is used to meticulously annotate various anomalies, not only marking their location and scope but also considering information such as severity, providing accurate labeled data for model training.

[0117] In an optional embodiment, in an input module of the fault detection model, the input size of the fault detection model is determined according to the image size and a Mosaic data enhancement algorithm is used to optimize the small target detection capability, including:

[0118] Step S201: In an input module, the input size of the fault detection model is determined according to the image size.

[0119] In step S202 , an image selection model based on probability distribution is used to assign different selection probabilities to each image according to the image category label and feature importance, and the corresponding category images are selected for splicing in the Mosaic process according to the probabilities.

[0120] Step S203 , based on the affine transformation matrix and the content features of the image, automatically determine its rotation angle, scaling ratio, translation position, and center of gravity during stitching, rotate it by a preset angle and translate it to a preset position in the mosaic image.

[0121] In step S204 , for each adjacent image pair among the multiple images participating in the mosaic stitching, the weights of the pixels in the boundary area are calculated and pixel value fusion is completed.

[0122] Specifically, considering the characteristics of flip-chip packaging and electrical equipment images, the appropriate input size is determined based on the actual size of the image captured by the imaging device. A smaller size (such as 320×320×3) can be set to highlight small objects while avoiding excessive redundant information. The input module still uses the Mosaic data augmentation algorithm to stitch multiple images into a suitable size, enriching the dataset and enhancing the model's robustness and small object detection capabilities.

[0123] Furthermore, the appropriate input size is determined according to the image size obtained by the high-precision imaging device. The original high-precision imaging device obtains image I, the input size is WxH (width is W, height is H), and the input module outputs the input size as I w xI h , select I w and I h Satisfy I w xI his a smaller value, and highlights small target features with better resolution (e.g. ).

[0124] The input module still uses the three-step innovation method of the Mosaic data enhancement algorithm. In the Mosaic data enhancement algorithm, let the original image be I and the number be n", and the original image set be {I1, I2, ... I n}, the size of a single image is wxh, the stitched image I mosaic Size is W mosaicx xH mosaic , where W mosaicx H is the width of the stitched image. mosaic is the height size of the stitched image. i Part of the area is copied to I mosaic The appropriate position to ensure that after splicing I mosaic The size is 640x640, then W mosaicx =640,H mosaic =640, satisfied (where n is determined by the specific stitching method), multiple images are stitched into a suitable size to enrich the dataset and enhance the model robustness and small target detection capabilities.

[0125] Mosaic's innovative algorithm employs a three-step approach. First, a probability-based image selection model assigns each image a different probability of selection based on factors such as its category label and feature importance. For an image classification task, images in different categories, particularly rare categories, are given a higher probability of selection. Specifically, the probability is determined by the inverse of their proportion, and the selection probability is calculated for all defects. These probabilities are then used to select images of the corresponding categories for stitching during the Mosaic process.

[0126] Assume that the image dataset has n categories, denoted as C1, C2, ...C n , the number of images corresponding to each category are m1, m2, ...m n , the total number of images is For Category C i , the probability of being selected is P(C i )for: In the Mosaic process, we can use the above probability P(C i ) to select from the corresponding category C by random sampling, etc. i Select images for stitching operation to realize image selection based on probability distribution.

[0127] The second method is to automatically determine the rotation angle, scaling ratio, translation position, center of gravity, etc. of the image during stitching based on the content characteristics of the image based on the affine transformation matrix, so that it can be rotated to the appropriate angle and translated to the appropriate position in the mosaic image that can highlight the main features.

[0128] Take the two-dimensional affine transformation as an example:

[0129]

[0130] This 3×3 matrix is ​​a two-dimensional affine transformation matrix, which functions as follows: (1) Linear transformation part: a and b: control the transformation in the horizontal direction (x-axis direction). Among them, a affects the scaling and shearing in the horizontal direction, and b affects the shearing in the vertical-horizontal direction. d and e: control the transformation in the vertical direction (y-axis direction). e affects the scaling and shearing in the vertical direction, and d affects the shearing in the horizontal-vertical direction. (2) Translation part: c and f: control the translation amount in the x-axis and y-axis directions respectively. In homogeneous coordinates, by applying matrix multiplication to a two-dimensional point (x, y, 1), the affine transformation of the point can be achieved, including operations such as rotation, scaling, shearing, and translation, thereby changing the position, direction, and shape of the object in the image.

[0131] The image is rotated by angle θ, translated by units tx along the x-axis, and translated by units ty along the y-axis. The corresponding affine transformation matrix is ​​as follows:

[0132]

[0133] For each pixel coordinate (x, y) in the image, by multiplying it with this transformation matrix (in homogeneous coordinate form, (x, y) is expressed as (x, y, 1)), the transformed coordinate position can be obtained, thereby realizing spatial transformations such as image rotation and translation, which are applied to flexibly layout images during Mosaic stitching.

[0134] The third method is based on image fusion:

[0135] Definition of model related variables: Assume that there are n images involved in Mosaic splicing, denoted as I i (i=1,2,…n), image I i The height is H i , width W i For any two adjacent images I m and I n (m, n∈{1,2,…n} and m≠n), the width of the splicing boundary between them is set to W mn (The width of the boundary between different adjacent image pairs may be different, and is given here as a general representation).

[0136] For any pixel point P(x, y) in the stitching boundary area (the range of x and y depends on the size of the image), define its pixel point P(x, y) to image I m The distance between the centers is d m (x, y), to image I n The distance between the centers is d n (x, y), image I m The weight at the pixel fusion is W m (x, y), image I n The weight at the pixel fusion is W n (x, y). Let the pixel value of the pixel in the kth color channel (such as the common RGB three channels, k = 1, 2, 3 corresponds to different channels) after fusion be F k ((x, y), image I m The pixel value of the kth color channel at the pixel point (x, y) is I mk ((x, y), image I n The pixel value of the kth color channel at the corresponding relevant position (the position considering the splicing correspondence) is I nk (x, y).

[0137] Weight calculation: For adjacent images I m and I n The pixel point P(x, y) in the stitching boundary area (the pixel point P(x, y) in the stitching boundary area should be determined based on the specific boundary width W mn and image width, etc., when x satisfies a specific range condition related to the corresponding image width and border width, it is determined to be at the border), and the distance is as follows:

[0138]

[0139] The weight calculation formula is:

[0140]

[0141] Calculation of pixel value after fusion: For the pixel point P(x, y) in the stitching boundary area, the general formula for calculating the pixel value in the kth color channel after fusion is:

[0142] F k (x, y) = W m (x, y)xI mk (x, y) + W n (x, y)xI nk (y,x-(W m -w mn )), k=1,2,3

[0143] Based on this general mathematical model, the weights of the pixels within the boundary regions of each pair of adjacent images participating in the mosaic are calculated and the pixel values ​​are fused. This results in a more natural transition between the stitched edges, enabling mosaic optimization based on image fusion, improving overall visual quality and the performance of subsequent applications. In actual applications, details such as boundary determination criteria and image correspondence can be further refined based on specific needs.

[0144] In an optional embodiment, the SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability, and further includes:

[0145] Step S205: dynamically adjust the window size according to the size of the input feature map.

[0146] Step S206 : adaptively and dynamically adjust the number of windows along the height direction and the width direction according to the step size parameter.

[0147] Specifically, the basic module (Backbone) is improved as follows: Convolution kernel size and stride adjustment: In the initial convolution layer, the convolution kernel size is adjusted to 3×3 or 4×4, and the stride is set to a value between 0.5-1. This can more finely extract the features of small targets (such as solder balls, protrusions, etc.) in flip-chip packages, ensuring that there are enough sampling points in the small target area. Depending on the actual situation, it is possible to consider introducing other modules or structures suitable for small target feature extraction, such as lightweight convolutional neural network modules, to further enhance the model's perception of small targets. For example, in subsequent convolution layers, the convolution kernel size and stride are dynamically adjusted according to the feature map size and target scale. For small feature maps close to the target size, a convolution kernel of 3x3 and a stride of 1 are used. The SW-MSA module is introduced in the Backbone module to enhance the model's ability to extract small target features.

[0148] Furthermore, the input image of the initial convolutional layer is F0, whose size is W0xH0xC0, where C is the number of channels. The convolution kernel size is set to k init (k init ∈{3x3, 4x4}), with a stride of s init (s init ∈[[0.5,1]), the size of the output feature map F1 after convolution is C1 is the new channel number.

[0149] In the subsequent convolutional layer, let the current feature map be F cur , whose size is W cur xH cur xC cur , whose target is T scale The image size is F w xFh , the target size is T scale =T w xT h ,when and (∈ is a small threshold, ∈ = 5), the convolution kernel size k = 3x3, the stride s = 1; the convolution kernel size and stride are dynamically adjusted according to the feature map size and target scale. For small feature maps close to the target size, a convolution kernel of 3×3 and a stride of 1 are used. The SW-MSA module is introduced into the Backbone module to enhance the model's ability to extract small target features. The feature map input to the SW-MSA module is Its dimensions are NxCxHxW (N is the batch size). SW-MSA output module Divide X into multiple overlapping windows, each of size MxM, M∈{3,4}, perform multi-head attention calculation in each window, let the number of heads be h, for each head I, (i=1,2,…,h), calculate the attention score A i , Y = SW-MSA(X), and then concatenate the results of all heads to obtain the output feature map Y.

[0150] In the embodiment of the present invention, F cur (Current feature map): This is the feature map processed by the current convolution layer. Its dimensions are three-dimensional, namely width (W cur ), height (H cur ) and the number of channels (C cur ). scale (Target scale): The size that the feature map is expected to reach, expressed as T w (target width) and T h (target height). w xF h : The actual size of the current feature map, where F w Equivalent to W cur , F h Equivalent to H cur . T w ×T_h: target size, that is, the size we want the feature map to reach after the convolution operation. The width F of the current feature map w With target width T w After the difference is rounded up, its absolute value does not exceed the threshold ∈ (here ∈ is set to 5). This means that the width of the current feature map is close to the target width. The height F of the current feature map h and target height F hAfter the difference is rounded up, its absolute value does not exceed the threshold ∈ (also ∈ is 5). This indicates that the height of the current feature map is also close to the target height. and When these two conditions are met, it means that the size of the feature map is close to the target size. At this time, a smaller convolution kernel (3×3) and stride (s=1) will be used for the convolution operation, so as to avoid excessive reduction of the feature map size while retaining the detail information. In other cases, the appropriate convolution kernel size and stride will be dynamically selected according to the specific situation of the feature map and the target scale. This strategy of dynamically adjusting the convolution kernel size and stride is mainly to adapt to feature maps of different sizes. For small feature maps close to the target size, using small convolution kernels and small strides can better retain the detail information of the image and help the model extract small target features. At the same time, combined with the SW-MSA module, it can further enhance the model's attention to and processing capabilities of small target features, and improve the model's performance in tasks such as target detection. Assume that the current feature map size is 100×100, the target size is 95×95, and the threshold ∈ is 5. Calculate the difference at this time: 5≤5, meets the conditions; 5≤5 also meets the condition. Therefore, in the subsequent convolution operation, a 3×3 convolution kernel and a stride of 1 will be used.

[0151] Input feature map processing and window division

[0152] The feature map input to the SW-MSA module is Its dimensions are NxCxHxW (N is the batch size). The window size M is no longer limited to a fixed {3, 4}, but is dynamically determined according to the size of the input feature map.

[0153]

[0154] k is an adjustable parameter (adjustable during training based on validation set performance, with an initial value of 10, for example). This allows the window size to better adapt to input feature maps of varying sizes, enhancing adaptability. This dynamic adjustment method adaptively selects the appropriate window size based on different input feature maps, resolving the existing problem of fixed, unadjustable input windows.

[0155] Divide X into multiple overlapping windows, and the number of adaptive dynamic windows that can be divided along the height direction The number of adaptive dynamic windows that can be divided along the width direction Here, s is a step parameter (0 < s ≤ M, and it can be set to M / 2 for experiments. This way of re - viewing can better capture the relationships between features. The step parameter s is used to divide the window, making the windows overlap. This division method is different from the traditional non - overlapping or fixed - pattern division, which can better capture the relationships between features and make full use of the information in the feature map. The model can better capture the complex relationships between features and generate more representative feature maps. This helps with subsequent detection and diagnosis tasks, such as more accurately identifying tiny fault defect patterns, reducing misjudgment and missed - judgment situations, and improving the reliability of the entire fault defect detection and diagnosis system.

[0156] Determine the appropriate input size according to the image size of the imaging device and use the Mosaic data augmentation algorithm to highlight small - target features and enrich the dataset, enhancing the model's robustness and small - target detection ability. Flexibly adjust the convolution kernel size and stride, including dynamically setting them according to the feature map situation in the initial and subsequent convolution layers, and also introduce the SW - MSA module to effectively improve the ability to extract small - target features. Enhancement of small - target detection ability: A series of optimization adjustments from the input module to each sub - module make the model more accurate in extracting features of small targets (such as protrusions, solder balls, etc.) in flip - chip packaging, and the detection ability is significantly improved, which is reflected in indicators such as the recall rate of small targets.

[0157] In an optional implementation manner, multi - scale feature fusion optimization and spatial pyramid pooling adjustment operations are performed on the neck module of the fault detection model, including:

[0158] Step S207: Increase the number of upsampling and downsampling times and set multiple sizes for feature map fusion.

[0159] Step S208: Adopt multiple aggregation strategies and assign different weights according to the importance of different - scale feature maps for small - target detection.

[0160] Step S209: In the spatial pyramid pooling part, set pooling operations with multiple spatial sizes.

[0161] Step S210: In the concatenation operation after pooling, adjust the mapping method of the number of channels according to the feature richness of the image and the number of targets.

[0162] Specifically, the neck module (Neck) was improved as follows: Multi-scale feature fusion was optimized, with increased upsampling and downsampling. In addition to the original fusion of feature maps of three sizes (20×20×1024, 40×40×512, and 80×80×256), smaller sized feature maps (such as 10×10×2048) with more detailed information were added. Different aggregation strategies, such as weighted fusion, were employed to more finely integrate feature maps of different resolutions by assigning different weights based on their importance to small object detection.

[0163] The following improvements were made to the Spatial Pyramid Pooling (SPPF): In the Spatial Pyramid Pooling part, a 3×3 pooling operation was added to the existing 1×1, 5×5, and 9×9 max pooling operations to better adapt to the feature extraction of small objects. Furthermore, in the post-pooling concatenation operation, the channel mapping method was appropriately adjusted based on the feature richness and number of objects in the flip-chip package image to avoid information loss or over-compression.

[0164] The Neck module optimizes multi-scale feature fusion. Specifically, it adds smaller feature maps rich in detail, such as 10x10x2048, to be fused with the original 20x20x1024, 40x40x512, and 80x80x256 feature maps. The fusion order and method are determined through experiments, and a weighted fusion strategy is adopted, assigning higher weights to small feature maps and appropriate weights to large feature maps based on their contribution to the target context. The details are as follows:

[0165] Multi-scale feature fusion optimization:

[0166] Let the newly added small-size feature map be F new , for example, adding a smaller feature map with rich detail information such as 10×10×2048. The original feature map is F 20x20 、F 40x40 、F 80x80 , such as the original 20×20×1024, 40×40×512, 80×80×256 features. The fused feature map F fuse The calculation is as follows:

[0167] F fuse =w new F new +w 20x20 F 20x20 +w 40x40 F 40x40 +w 80x80 F 80x80

[0168] Among them, w new =0.5, w 20x20 =0.3, w40x40 =0.15, w 80x80 = 0.05. These weights can be further adjusted based on experiments and during the fusion process, different feature maps are upsampled or downsampled to match their sizes.

[0169] In the splicing operation after pooling, the mapping method of the number of channels is adjusted according to the feature richness and target number of the chip package image. Let the input feature map of the SPPF module be F in , the feature maps obtained after 1x1, 3x3, 5x5, and 9x9 pooling operations are F 1x1 、F 3x3 、F 5x5 、F 9x9 . Let the weight vector be w=[w 1x1 ,w 3x3 ,w 5x5 ,w 9x9 ], where w 3x3 =0.4, w 1x1 =0.2, w 5x5 =0.3, w 9x9 =0.1.

[0170] The fused feature map F pooled =w 1x1 xF 1x1 +w 3x3 xF 3x3 +w 5x5 F 5x5 +w 9x9 F 9x9 In terms of channel number mapping, let the mapping function M be F pooled The number of channels from C pooled Mapping to C out , that is, F pooled =M(F poed ), the mapping method can be a simple linear transformation or other appropriate functions.

[0171] For example, in the pooling operation adjustment, a 3×3 pooling operation is added to the SPPF module, which is combined with the existing 1×1, 5×5, and 9×9 maximum pooling operations to extract global features at different scales. The weights or proportions of the different pooling operations are appropriately adjusted based on the characteristics of small objects, and the channel number mapping method is redesigned in the post-pooling concatenation operation to ensure that small object features are effectively represented in the channel dimension.

[0172] Multi-scale feature fusion optimization adds smaller feature maps rich in detail information and adopts a weighted fusion strategy. At the same time, new operations are added to the pooling operation and the weights are adjusted to ensure that small target features are effectively represented in the channel dimension.

[0173] In an optional embodiment, adjusting the attention mechanism of the SW-MSA module in the prediction module of the fault detection model includes:

[0174] Step S211, calculate the target according to the self-attention in the window and adjust the window size.

[0175] Step S212 , adjusting the number of heads in the multi-head mechanism according to target feature distribution of chip packaging and electrical devices.

[0176] Specifically, the following improvements are made to the prediction module (Head): Attention mechanism adjustment: In the SW-MSA module, the window size is adjusted to a 3×3 or 4×4 window so that the self-attention calculation within the window is more focused on small targets. According to the target feature distribution of the flip-chip package, the number of heads in the multi-head mechanism is adjusted to optimize the feature capture of different subspaces. The following improvements are made to the global information fusion module: The parameters of the linear layer are adjusted so that the mapped query (Q), key (K), and value (V) matrices can better reflect the global and local relationship of the small target, so as to more accurately fuse the global and local features and meet the high accuracy requirements of flip-chip package detection. Histogram equalization, small angle rotation (such as -15° to 15°), small scale scaling (such as 0.8-1.2 times).

[0177] Attention mechanism adjustment In the SW-MSA module, the window size M∈{3,4} is set, and the number of heads h is determined according to the target feature distribution of the flip chip package, for example, h=8. For the linear layer in the global information fusion module, the input query matrix Q is set in , key matrix k in , value matrix V in , whose dimensions are Nxd q xHxW、Nxd k xHxW、Nxd v xHxW(d q d k d v (corresponding dimension) linear layer parameter matrix w Q 、w K 、w Y The dimensions are d q xd model d K xd model d y xd model (d model is the model dimension). Output matrix Q out =Q in w Q , K out =K in w k , V out =Vin w V , by optimizing w Q 、w k 、w V To better capture the global and local relationships of small targets.

[0178] For example, in the SW-MSA module, the window size is set to 3×3 or 4×4, and the number of heads in the multi-head mechanism is adjusted based on the target feature distribution of flip-chip packaging. The linear layer parameters in the global information fusion module are optimized so that the mapped query (Q), key (K), and value (V) matrices better reflect the global and local relationships of small targets.

[0179] In an optional embodiment, step S3 includes:

[0180] Step S301: During the chip package fault detection model training process, the training data set is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the performance indicators of the model on the validation set under different batch values.

[0181] Step S302: gradually increase the number of training rounds according to the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance index meets the preset conditions.

[0182] Step S303: Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate.

[0183] Step S304: adopt a learning rate decay strategy and adjust the decay rate and period according to the performance indicators of the model on the validation set.

[0184] Step S305: linearly transform the feature vectors in the window to obtain query vectors, key vectors and value vectors through the adaptively learned weight matrix. The weight matrix is ​​adaptively adjusted according to the back propagation of the loss function during the training process.

[0185] In step S306, a learnable bias matrix is ​​introduced for each head, and multi-head attention calculation is performed in each window to obtain an attention score.

[0186] Step S307: Perform weighted summation on the attention score and value vector to obtain a single-head output feature vector.

[0187] In step S308, the output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

[0188] Specifically, batch size adjustment: Let the batch size be b (b∈[8,16]), for the training data set D, it is randomly divided into several batches {B1,B2…,B m} The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in indicators such as accuracy Acc, recall Rec, and mAP-50 on the validation set under different b values.

[0189] Adjustment of the number of training rounds: Assume that the number of training rounds of the original plan is E original , the adjusted number of training rounds E = (1 + r)E original (r∈[0.2,0.5]). In each round, performance indicators such as Acc(e), Rec(e), and mAP-50(e) are calculated (e represents the current round number during training, and the accuracy of the model on the validation set is calculated). Training stops when the following conditions are met:

[0190] |Acc(e)-Acc(e-1)|<δ acc and |Rec(e)-Rec(e-)|<δ rec And |mAP-50e-mAP-50e-1<δmap

[0191] Among them, δ acc , δ rec , δ map is a small threshold, such as δ acc c=0.001,δ rec e=0.001,δ map P = 0.001. Example: Model Training Parameter Adjustment Learning Rate Adjustment: Select an initial learning rate between 0.0001 and 0.001. Experiment with small-scale datasets to observe model convergence under different initial learning rates and determine the optimal value. Adopt a learning rate decay strategy (e.g., lower cosine annealing), adjusting the decay rate and period based on the model's performance on the validation set. By properly adjusting parameters such as the learning rate, batch size, and number of training rounds, the model training process becomes more stable, fully utilizing sample feature information and avoiding averaging of small target features. Ultimately, better results are expected in performance metrics such as accuracy, recall, and mAP-50.

[0192] Adaptive multi-head attention calculation (for each head i): For each head i (i=1,2,···,h), multi-head attention calculation is performed within each window.

[0193] Linear transformation: Let the eigenvector x in the window m，n (located in the window of row m and column n), the weight matrix learned through adaptive learning (Here we assume that C is divisible by the number of heads h) and perform a linear transformation to obtain the query vector Qi , key vector K i Sum value vector V i , whose values ​​are as follows:

[0194] Q i =x m，n W Qi ;K i =x m，n W ki ; V i =x m，n W Vi

[0195] Here the weight matrix W Qi 、W ki 、 During training, adaptive adjustments are made based on the backpropagation of the loss function to improve accuracy. In the SW-MSA module, the window size and the number of multi-head mechanisms are adjusted, and the linear layer parameters are optimized to better reflect the global and local relationships of small objects.

[0196] Calculate the attention score A i :

[0197] Among them, d k =C / h, E i is a learnable bias matrix (with dimensions By introducing a learnable bias, the model can better capture complex feature relationships and improve accuracy.

[0198] According to the attention score A i , and the value vector V i Get the single-head output feature vector

[0199] The summation here is performed on all positions in the window to obtain a value that integrates the attention information. Adopting the adaptive multi-head attention mechanism, in the linear transformation stage, the weight matrix W Qi 、W Ki 、W Vi During training, adaptive adjustments are made based on the backpropagation of the loss function. This enables the model to automatically learn appropriate weights based on the characteristics of the data, improving its adaptability to diverse data. Introducing a learnable bias matrix when calculating attention scores can better capture complex feature relationships, enabling the model to more accurately focus on important features. This is an improvement over traditional multi-head attention calculations.

[0200] Concatenate the output feature maps Y of all heads i=1,2,…,h along the channel dimension to obtain the final output feature map Y, Y=concat(Y 1,Y 2 ,...,Y h ). After splicing That is, Y = SW - MSA(X); the output feature map, symbolized as Y, is obtained by concatenating the outputs Yi from all heads. This design is consistent with the standard multi-head attention mechanism. By processing multiple heads in parallel and fusing the results, the model can capture feature information from different subspaces, improving expressiveness.

[0201] During the training process, a high-precision loss function is used, such as a composite loss function that takes into account position error, category error, and small target feature matching, to optimize model parameters by minimizing them to meet the evaluation criteria of high accuracy, high precision, and high adaptability. For example:

[0202] L=L lOC +L Cls +Enter L feat , where L lOC is the position loss (such as loU loss, etc.), L Cls is the category loss (such as cross entropy loss), L feat is the small target feature matching loss (which can be defined by comparing the difference between the predicted small target features and the true small target features), and is the balancing parameter.

[0203] The composite loss function optimization uses a compound loss function that simultaneously considers location error, classification error, and small target feature matching. This composite loss function more comprehensively evaluates the discrepancy between the model's predictions and the ground truth, and is more effective in optimizing model performance across multiple dimensions than a single loss function. By comprehensively considering multiple errors, the composite loss function further optimizes the model's predictions, enabling the model to more accurately locate the location and classification of faults.

[0204] The innovative technical effects of the improved SW-MSA module include: improved adaptability; dynamic window sizing and adaptive step-size window partitioning enable the model to adapt to input feature maps of varying sizes. The model can effectively process both large and small flip-chip multi-layer micron chip defect image data, enhancing the model's adaptability to varying data sizes and feature distributions. The learnable weights and biases in the adaptive multi-head attention mechanism more accurately capture small object feature information, improving the model's ability to extract small object features, thereby enhancing the model's accuracy and precision in detecting flip-chip multi-layer micron chip defects.

[0205] In an optional implementation, step S3 further includes:

[0206] Step S311: During the training of the fault detection model for electrical equipment, the training data set is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the performance indicators of the model on the validation set under different batch values.

[0207] Step S312: gradually increase the number of training rounds according to the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance index meets the preset conditions.

[0208] Step S313: Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate.

[0209] In step S314, a small target detection penalty term is added based on CIOU_LOSS, and a weight coefficient is determined according to the ratio of target size to image size to increase the weight of small target position and category error. At the same time, FocalLoss is used to adjust the adjustable parameters to solve the category imbalance problem.

[0210] Step S315 , introducing multiple evaluation indicators to evaluate the performance of the model in flip-chip electrical device detection, so as to adjust and optimize the model.

[0211] In step S316, for each head, a learnable bias matrix is ​​introduced, and multi-head attention calculation is performed in each window to obtain an attention score.

[0212] Step S317: Perform weighted summation on the attention score and value vector to obtain a single-head output feature vector.

[0213] In step S318, the output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

[0214] Specifically, let the learning rate be l r , the number of training rounds is t, and the initial learning rate is l r0 , the minimum learning rate is l rmin , the total number of training rounds is T.

[0215] The cosine annealing learning rate adjustment formula is

[0216] Parameter range: initial learning rate l r0 The value is between 0.0001-0.001, and the minimum learning rate is l rmin Can be set according to specific circumstances (for example, it can be set to l rmin =0.00001 or other smaller values), the total number of training rounds T is determined according to the performance change of the model on the validation set, and is calculated by multiplying the number of training rounds by the number of steps (batches) in each round. The number of training rounds t starts from 0 and gradually increases to T during training.

[0217] Batch size adjustment: Set the batch size to 8-16, compare the stability and performance improvement of model training under different batch sizes to determine the optimal value, so that the model can fully utilize the feature information of each sample and avoid averaging of small target features.

[0218] Adjustment of the number of training rounds: In the original training rounds rorignal , the adjusted number of training rounds round new =(1+r)round rorignal , where r ranges from 20% to 50%. During training, monitor the changes in performance metrics such as accuracy, recall, and mAP-50 on the validation set. The optimal number of training epochs is determined by monitoring the model's accuracy, recall, and other performance metrics on the validation set. This ensures that the model fully learns the complex features of small objects. We have made targeted adjustments to the learning rate, batch size, and number of training epochs, and determined the optimal values ​​through experimentation to ensure that the model fully learns the complex features of small objects.

[0219] Optimize the loss function and introduce new evaluation indicators, such as adding a small target detection penalty term based on CIOU_LOSS: the loss function is Loss, CIOU_LOSS is L CIOU , the target size is Starget, the image size is Simage, and the weight coefficient is w (determined according to the ratio of target size to image size), then Loss = LcIou + wxpenaltysmall_target (where penaltysmall_target is the penalty term for small target detection, and its specific form is designed according to the small target detection problem).

[0220] Combined with Focal Loss: Let the category prediction probability be p, the true category be y, and the Focal Loss be L FOCAl , γ and α are adjustable parameters (through experimental adjustment, the model can pay more attention to rare adverse situations, such as γ can be set between 2 and 5, and α can be set between 0.25 and 0.75 according to the category imbalance), then L FOCAl =-α(1-pγylog(p)-1-pγp(1-y)log(1-p), the final loss function is Loss=LCIOU+w xpenaltysmall_target+L FOCAl .

[0221] Based on CIOU_LOSS, a penalty term for small object detection is added. The weight coefficient is determined based on the ratio of the object size to the image size, increasing the weight of small object position and category errors. Focal Loss is also used, and the γ and α values ​​are adjusted to address the class imbalance problem, making the model more sensitive to rare adverse conditions (such as tiny holes).

[0222] Loss function optimization

[0223] Let the target box be Bgt, the predicted box be Bp, and the intersection and union ratio be Center coordinate difference △x=x gt -x p ,△y=y gt -y p , the width and height of the target frame are w gt 、h gt , the predicted box width and height are w p 、h p , CIOU loss Where p is the Euclidean distance and c is the minimum distance between two boxes.

[0224] The length of the diagonal of the enclosed region, Small target detection penalty Lpenaly: set the target size Image size when (θ is a threshold value, such as 0.1), Lpenaly=λ(|△x|+|Δy|+|w gt -h gt |+|h gt -h p ,|)(λ is the weight coefficient, determined according to experiments, such as 0.5).

[0225] Focal Loss FL(p t )=-(1-p t ) γ log(p t )γlog(pr) Open the bean bag question, let the probability of the positive class be p, the probability of the negative class be 1-p, p t =p(positive class) or p t =1-p(negative class), adjust γ=2, a=0.25.

[0226] Evaluation indicators for weak target detection with high noise levels: Design ideas for indicators related to the computational mathematical model for evaluating anchor frame quality in micron chip defect and fault detection.

[0227] 1. Define relevant variables

[0228] Let the outlier value of the anchor box quality measure be β, the non-monotonic focusing factor be r, the aspect ratio of the sample be a, the centroid distance be d, and the overlapping area be s.

[0229] 2. Construction of the relationship between the non-monotonic focusing factor r and the outlier β

[0230] The functional relationship between r and outlier β is r = f(β), and the smaller the β value, the smaller the r. A simple inverse proportional form is constructed. (∈ is a constant, ∈ is a very small positive number to avoid the denominator being).

[0231] 3. Anchor Box Weight Setting in Loss Function

[0232] The weight of the high-quality anchor box in the loss function is w h , the weight of the low-quality anchor box in the loss function is w l , according to the rule: high-quality anchor boxes correspond to smaller β, and their weight w h Negatively correlated with r (because the smaller r is, the higher the weight of the high-quality anchor box is), we can set; w h =1-r. Low-quality anchor boxes correspond to larger β, and their weight w l Positively correlated with r (the larger r is, the lower the weight of low-quality anchor boxes is, which reduces their impact), we can set w l =r.

[0233] IV. Overall Performance Evaluation Indicators

[0234] Suppose the overall performance evaluation index of the model is P, which is affected by the average quality sample. The relevant features of the average quality sample can be expressed by comprehensively considering the aspect ratio a, the centroid distance d, the overlapping area s, etc. The comprehensive value of the relevant features of the average quality sample is Q(a, d, s) (the specific function form is weighted summation, etc. Q(a, d, s) = α1a + α2d + α3s, α1, α2, α3 are the corresponding weight coefficients). Considering the influence of the weights of high-quality and low-quality anchor boxes on the overall performance, the calculation model of P is constructed as follows: P = g(w h , w l , Q(a, d, s)), can be constructed as: P = w h. Q(a, d, s)-w l (1-Q(a, d, s)).

[0235] Through this mathematical model, β is used to dynamically adjust r, thereby rationally allocating the weights of high-quality and low-quality anchors in the loss function. This, combined with the sample's own characteristics, comprehensively evaluates the model's overall performance, achieving the goal of rationally evaluating anchor quality and measuring model effectiveness in micron chip defect and fault detection. This solution can be used to detect and track defects and visualize faults in large products such as transformers, inverters, converters, capacitors, cables, and IGBTs.

[0236] This paper provides a high-precision, highly adaptable intelligent detection and diagnosis method for electronic product defects. Taking into account the characteristics of flip-chip package images, the method determines the appropriate input size based on the actual size of the image captured by the imaging device to highlight small object features. The input module also uses the Mosaic data augmentation algorithm to stitch multiple images into a suitable size, enriching the dataset and enhancing the model's robustness and small object detection capabilities.

[0237] The present invention provides a high-precision and high-adaptability intelligent detection and diagnosis system for electronic product defects, comprising: an actively excited infrared imaging zoom camera, an industrial-grade CMOS sensor, an excitation unit, a laser driver, a CW laser, a reflector, a programmable logic controller, a cylindrical lens, a wire harness exciter, and a heat spreader.

[0238] An active-excitation infrared imaging zoom camera captures thermal images of chip packaging and electrical equipment, and divides the thermal images into a training learning image group and a test detection image group in proportion; images of defective faults and non-defective faults in the two groups are added to the YOLOV5 algorithm system respectively; the thermal images are classified by product, and then the classified images are used to implement the training learning set and detection test set of the YOLOV5 algorithm respectively; after the quality of the training learning set and the detection image meet the performance evaluation indicators, the images are inspected to identify the fault defect location and classification.

[0239] Specific, combined Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7As shown, the embodiment of the present invention adopts an active excitation infrared imaging (AEII) fault defect detection, diagnosis and tracking device and a YOLOV5-CDF-SW-MSA intelligent algorithm fusion system for application in the assembly, detection and use process of current, voltage, and frequency sampling sensors and power electronic products in the power system, and progressive aging faults and welding images of PCB boards and flip chips (including multi-layer chips, micron-level multi-layer chips, metering chips, main control chips (MCU chips), communication chips, security encryption chips (ESAM chips), clock chips, storage chips, RF front-end chips, etc.). The images should contain various possible states, including normal state, missing protrusions, cracks, voids, missing balls, cold solder joints, surface contaminants, internal contamination, packaging material aging, solder composition changes, electrical short circuits, resistance abnormalities, loose packages, component displacement, abnormal solder balls in key areas, defects in different areas (such as defects in the upper left corner area and center area), mounting offsets, solder short circuits, potential cracks, potential cold solder joints, and other defects and material and structural fault location detection, diagnosis and tracking.

[0240] The embodiments of the present invention are not limited to active-excitation infrared imaging zoom cameras, nor are they limited to the following technical requirements: infrared detection areas encompassing near-infrared (0.75-1.5 μm), mid-infrared (1.5-20 μm), and far-infrared (20-1000 μm); a maximum frame rate in full-frame mode; and a high-speed, high-resolution thermal infrared camera frame rate. The imaging chip must have flexible pixel reading and processing methods, high pixel density, and fast readout capabilities, supporting a high frame rate of 200 frames per second and multiple resolutions. Industrial-grade CMOS sensor; resolution mode is switched by setting different pixel merging or reading methods inside the sensor; 4K ultra-clear during the day and switching to full-color lens at night, dynamic range 200Db, pixel depth 16bit, sensitivity less than 0.01Iux (back-illuminated (BSI) CMOS sensor with sensitivity less than 0.01lux, large-size photosensitive element and low-noise signal amplification circuit, while pixel depth 16bit and dynamic range 200dB, lens matching with high-transmittance optical lens to reduce light loss, so as to make full use of weak light for imaging), C-mount, F-mount, resolution (black and white, color, near-infrared enhancement, simultaneous focus on multiple levels at the same time, adaptable to lenses of various focal lengths and apertures, focal length range from wide angle (2mm) to telephoto (120mm) to facilitate zoom function, lens and camera combination based on phase detection autofocus (PDAF) technology, high-precision autofocus on multiple levels of focus system), resolution in black and white mode generally needs to reach More than 20 million pixels, such as 2448×2048; color mode: for visual analysis of power electronic products, the resolution in color mode is usually not less than 4K, that is, 3840×2160; near-infrared enhanced mode: this mode is mostly used for special detection needs, such as detecting the internal structure of the chip, and the resolution is generally 1920×1080; signal-to-noise ratio is above 60dB; and the excitation unit AWG modulation voltage (providing a stable voltage output of several volts to tens of volts), the laser driver modulation current (the laser driver modulation current part depends on the laser type and power, sufficient current output capacity, stability and accuracy, Thorlabs), CW lasers such as using arbitrary waveform generators (Keysight 33600A), the reflector and DOE are determined according to the laser optical path design and the required beam shaping effect, PCL (synchronous triggering, different waveform control, etc.) controls the multi-point thermal wave laser to form the excitation mechanism to trigger the writing lens, and the control computer controls the excitation unit.When it comes to high-speed thermal infrared cameras and close-up cameras (for close-up shots and other application scenarios, wavelength, power and beam quality need to be considered), and when a multi-point pulse linear laser generator is selected; the laser driver is selected based on the power and current requirements of the continuous wave laser, and the current output capacity must match the working current of the laser, and it must have high stability and precision, and the current regulation accuracy can reach the milliampere level or even higher; the optical component selection is cylindrical lens, linear beam generator, and heat spreader; the selection of cylindrical lens stretches or compresses the laser beam in one direction, so it is necessary to select a cylindrical lens with suitable focal length and size; the linear beam generator needs to consider the parameters such as the beam density and angle it generates to meet the detection requirements; the heat spreader needs to be selected based on the heat conduction efficiency and temperature uniformity, such as metal materials with high thermal conductivity (such as copper alloy); the near-infrared band is used for some special detections (such as chip internal structure detection), the control computer: has a high-speed data transmission interface (such as USB 3.0, PCI-E, etc.); multiple communication interfaces (such as RS-232, Ethernet, etc.); a high-performance workstation, such as the Precision series, with a powerful multi-core processor (such as the Intel Core i9 series) and large memory (such as 64GB and above).

[0241] The heat source selected by the active excitation infrared imaging zoom camera of the present invention is not limited to pulsed thermography (PT), lock-in thermography (LT), ultrasonically stimulated vibration thermography (UVT) and eddy current thermography (ECT).

[0242] Single-line laser lock-in thermal imaging is used to detect short circuits and defects in power transformers, cables, welding water, electricity, and gas meters, circuit breakers, conductive wire systems, circuit boards, PCBs, and welding. Multi-line laser lock-in thermal imaging is also used to detect short circuits and defects in composite materials and multilayer components in power transformers, cables, welding water, electricity, and gas meters, circuit breakers, conductive wire systems, circuit boards, chips, multi-layer chips, PCBs, and welding large-scale multi-layer welding. Single-line and multi-line laser lock-in thermal imaging is used. A continuous-wave laser beam is modulated into a pulsed laser beam by an excitation unit, and a cylindrical lens converts the pulsed laser beam shape from a point to a linear shape. The control unit then sends a control signal to a galvanometer scanner to direct the line laser beam onto the target surface. The line laser beam then generates thermal waves along the desired excitation line, scanning the target surface horizontally and vertically, effectively detecting randomly oriented cracks and other defects in real time. The integration of active excitation infrared imaging, zoom camera infrared sensing technology, and lock-in thermal imaging significantly improves the sensitivity and resolution of thermal imaging. The sensitivity of thermal imaging is improved by two orders of magnitude to about 102μK, while the resolution of surface and subsurface defects is reduced to 4.8μm. The 808nm optical tube combined with the present invention can detect micro-hole defects of 430um in the depth of 1.35mm deep delamination defects in PCB multi-layer complex structures.

[0243] Combine Figure 5 、 Figure 6and Table 1, the present invention utilizes the above configuration and conditions to actively excite infrared imaging zoom camera system, places the power transformer, cable, welding water, electricity and gas meter, circuit breaker, conductive wire system, circuit board, PCB board, PCB board and flip chip (including multi-layer chip, micron-level multi-layer chip, metering chip, main control chip (MCU chip), communication chip, security encryption chip (ESAM chip), clock chip, storage chip, radio frequency front-end chip, etc.) package image on the adjustable working detection table and carrier, infrared imaging zoom camera system and auxiliary close-up camera such as electron microscope (taking micron-level multi-layer chip as an example), active excitation infrared imaging zoom camera system takes infrared thermal image of welding defects of sampling component; data transmitted to the control system by the control system In the library, thermal images are divided into a training learning image group and a test detection image group in proportion; images of defect faults and non-defect faults in the two groups are added to the algorithm system based on the new YOLOV5 of the present invention respectively; thermal images are classified by product, such as the normal state, missing protrusions, cracks, voids, solder overflow, etc. of poor PCB welding of smart water, electricity and gas meters (including crystal oscillators, chips, etc.); chip defects include missed balls, cold solder joints, surface contaminants, internal contamination, aging of packaging materials, changes in solder composition, electrical short circuits, abnormal resistance, loose packaging, component displacement, abnormal solder balls in key areas, defects in different areas (such as defects in the upper left corner area, defects in the center area), mounting offset, welding short circuits, potential cracks, potential cold solder joints and other defects and material and structural failures. Then the above classified images are used to realize the training learning set and detection test set of the new YOLOV5 algorithm respectively; further, after the training learning set and the detection image quality reach the performance evaluation index output, the detection image, its fault defect location classification is accurately identified, such as the voids, cracks, protrusions, cold solder joints, delamination, impurities and other defects (including hidden minor defects) in micron chips, crystal oscillators, PCB board components and welding joints, two-dimensional and three-dimensional visualization of faults and products, providing a fault defect location diagnosis system, based on the high-precision, high-definition and high-adaptability infrared thermal imaging fault defect detection and diagnosis tracking device system for power electronic products combined with the new YOLOV5 algorithm for detection, realizing the chip package image and PCB board welding assembly detection and use, during the detection process, it can effectively detect and locate missing solder balls (lq), cold solder joints (xh), protrusion related annotations missing protrusions (mqxs), protrusion deformation (qtxb), lw (cracks), qlwy (solder overflow), tbqs (convex hull missing), tbzc (convex hull normal).The position and shape of lq (missing solder ball), xh (cold solder joint), mqxs (missing protrusion), qtxb (protrusion deformation), lk (crack), kd (void), bmwr (surface contaminant), nbwr (internal contamination), fbcl (aging of packaging material), hldf (change in solder composition), dqdl (electrical short circuit), dzcy (abnormal resistance), fbss (loose packaging), bjwy (component displacement), gjqy (abnormal solder ball in critical area), zjsq (defect in upper left corner area), zxqy (defect in center area), tzpy (mounting offset), hjdl (soldering short circuit), qzlk (potential crack), and qxh (potential cold solder joint) provide a basis for identifying fault anomalies, repairing solder joints and preventing cold solder joint process improvements, and provide methods for prediction and prevention during monitoring production and use. By monitoring the integrity of missing protrusions (mqxs) in real time, the chip's functionality, thermal performance, electrical performance, corrosion, contaminant type (e.g., organic, inorganic) and approximate coverage (e.g., localized, widespread), stability, and reliability are monitored. Internal contamination that could interfere with signal transmission within the chip or cause faults such as short circuits is detected, providing a means to assess abnormal signals within the internal structure, pre-discharge failure reliability life, discoloration, brittleness, and loss of adhesion. The severity and location of packaging material aging are noted, monitoring the characteristics of material aging. By detecting poor soldering due to changes in material properties, contamination-related conditions, material property changes, electrical performance, and mechanical stability are monitored. Combined with functional impact and location information, production processes, and potential issues, this is used to predict and prevent chip defects, service life, or potential reliability issues. This ensures the overall health of the PCB, integrated circuit, and chip throughout its lifecycle.

[0244] Table 1

[0245]

[0246] Application of the present invention Figure 6The result image also shows the different categories of targets detected by the model, as well as the confidence scores corresponding to each predicted bounding box. The high confidence score not only reflects the certainty of the model's prediction, but also further verifies the reliability of the YOLOV5-CDF-Mosaic-SW-MSA model of the present invention in the space infrared weak target detection task. By spot-checking 80 detection result images of micron multilayer chips, IGBTs, cables, transformers, and PCB boards used for smart meters, the performance of the YOLOV5-CDF-Mosaic-SW-MSA model in the space infrared weak target detection task is fully demonstrated (precision-recall curve rate multilayer chips 0.998, IGBT 0.989, cable 0.996, transformer 0.997, PCB board used for smart meters 0.997). The model not only performs well in different categories, but also maintains stable detection performance in various infrared image scenes. This reflects the robustness and strong generalization ability of the YOLOV5-CDF-Mosaic-SW-MSA model in processing space infrared weak targets. In summary, the detection images presented here provide an intuitive and comprehensive basis for evaluating the performance of the YOLOV5-CDF-Mosaic-SW-MSA model in the space infrared weak target detection task. These results not only demonstrate the model's efficiency and accuracy in detecting weak infrared targets, but also reveal its unique advantages in this type of task, providing a valuable reference for subsequent model optimization and application in YOLOV1-10 and other models.

[0247] Combining Table 1 with Table 2, the YOLOV5-CDF-Mosaic-SW-MSA model shows significant advantages in the field of target detection, especially in the detection of weak targets in spatial infrared. Through the effective combination of deep network structure and multi-scale feature fusion technology, the model has achieved significant improvements in target detection accuracy. Specifically, in the comparative analysis, the accuracy of YOLOV5-CDF-Mosaic-SW-MSA reached 99.96%, which is significantly better than other competing models such as the accuracy of YOLOV1 to YOLOV10 and 82.41% of Faster RCNN. This advantage is particularly evident when dealing with complex backgrounds and weak targets, extreme conditions and small object detection under extremely low lighting or severe occlusion, where existing models find it difficult to accurately identify and locate targets, and hidden defects and faults on the surface and sub-surface of objects are particularly obvious, such as Figure 3 As shown in the figure, the model of the present invention has achieved high detection accuracy in multiple categories, including the detection of infrared weak targets for defects and faults in micron multi-layer chips, IGBTs, transformers, and cables. The model can also be applied to YOLOV6 to YOLOV10.

[0248] YOLOV5-CDF-Mosaic-SW-MSA maintains a fast detection speed of 29ms while ensuring high accuracy. Its micron-based multi-layer chip achieves an image efficiency of 200 frames per second (FPS), meeting the needs of applications with high real-time requirements. Compared to other models, YOLOV5-CDF-Mosaic-SW-MSA combines high performance and high frame rate, providing strong support for practical applications.

[0249] Furthermore, through data augmentation and a stable training strategy, YOLOV5-CDF-Mosaic-SW-MSA demonstrates excellent robustness and stability. The model maintains consistent performance across a variety of scenarios and conditions, providing developers with a reliable solution. Furthermore, its open-source and modular design endows the model with excellent usability and scalability, offering developers broad application prospects.

[0250] By applying YOLOV5-CDF-Mosaic-SW-MSA to a wider range of tasks (such as video object tracking and 3D object detection), we can expand the application areas and functions of the model and further explore its potential. Through these efforts, we aim to further improve the performance of the YOLOV5-CDF-Mosaic-SW-MSA model and promote its development in the field of object detection.

[0251] Table 2

[0252]

[0253] Combine Figure 8 As shown, this invention improves upon YOLOV5-CDF-SW-MSA. Visual analysis plays a crucial role in different object detection tasks because it helps to intuitively understand model performance, identify potential issues, and evaluate the model's behavior in different scenarios. We will perform a detailed visualization analysis of the object detection results of our model, including a confusion matrix and visualization of the detection results.

[0254] First, the elements on the diagonal indicate that the model performs relatively well in most categories, with high true positive (TP) values, indicating that the model can accurately identify objects belonging to these categories. However, for some categories, the false negative (FN) values ​​are relatively high, indicating that the model has missed the detection of objects from these categories and needs further optimization to improve the recall rate.

[0255] Secondly, observing the off-diagonal elements, we found that the model showed some false positives (FP) in some categories. This may be due to the feature similarity between these categories and other categories, making it difficult for the model to accurately distinguish them. In order to reduce the false alarm rate, we can consider increasing the training samples of these categories and improving the applicability of different spaces and detection objects. In this paper, a more complex feature extraction method is adopted based on YOLOV5-CDF-SW-MSA to enhance the model's ability to distinguish categories, and the average false alarm rate is reduced by 0.0012.

[0256] In the result image ( Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 ), the invention shows the remarkable performance of the YOLOV5-CDF-Mosaic-SW-MSA model in identifying these spatial infrared weak targets (tiny targets and hidden targets). Each image carefully shows the predicted bounding box and prediction score. The predicted bounding box accurately locates the target position predicted by the model, and the prediction score quantifies the accuracy of the model in detecting the target. Figure 2 For example, in the first image on the lower left, the predicted bounding box of the infrared image of the passive NiCr resistor integrated on the flip-chip substrate (bad pin) closely hugs the true position of the target. "P: 0.996" indicates that the model has 99.3% accuracy in identifying the target as an aircraft. The bad abnormal temperature tracking has developed to 52°C-63°C. Figure 3 The crack accuracy on the multi-layer chip shown is 99.4%, the void accuracy is 99.9%, and the weld deficiency image lacks welds with an accuracy of 99.8%; Figure 4 Infrared images of chips with a 99.8% accuracy rate and crystal oscillator with a 99.9% accuracy rate on the PCB board that integrates power computing power; Figure 5 The tracking chip on the power transformer control PCB has an accuracy of 99.5% within the normal operating temperature range, but after running for 5 minutes, it will show an abnormal fault temperature of 75°C. Figure 6This paper presents graphs of precision (accuracy), recall, and confidence for detecting defects and faults on micron chips, IGBTs, transformers, and cables. Visual analysis of the area ratio of the precision-recall curve is performed: 0.989 for IGBTs, 0.998 for micron multilayer chips, 0.996 for cables, 0.997 for transformers, and 0.997 for PCB mAP@0.5. This demonstrates the model's high accuracy and reliability in detecting, diagnosing, and tracking these targets. Even though these targets exhibit low contrast and blurry features in infrared images, the YOLOV5-CDF-Mosaic-SW-MSA model is still able to accurately identify, locate, and track them. Notably, the detection accuracy for all images exceeds 996%, fully demonstrating the efficiency and accuracy of the YOLOV5-CDF-Mosaic-SW-MSA model for weak infrared target detection. Whether detecting, diagnosing, or tracking weak targets in infrared images, the model accurately identifies and tracks these weak targets, providing corresponding predicted bounding boxes.

[0257] Figure 7 The power product defect fault detection and diagnosis device and work flow shown:

[0258] 1. Device hardware system composition

[0259] This device consists of an active excitation infrared imaging system, an intelligent algorithm processing platform, and auxiliary hardware units. The functions of each part are as follows:

[0260] (1) Active excitation infrared imaging system

[0261] Infrared imaging module camera: supports multi-band detection of near infrared (0.75-1.5μm), mid-infrared (1.5-20μm), and far infrared (20-1000μm), equipped with an industrial-grade CMOS sensor, with resolutions covering 20 million pixels (black and white), 4K (color), and 1080P (near infrared enhanced), with a maximum frame rate of 200 frames / s, a dynamic range of 200dB, a pixel depth of 16 bits, a sensitivity of ≤0.01lux, and supports phase detection autofocus (PDAF) and zoom function (focal length 2-120mm).

[0262] Close-up: Used with an electron microscope (for micron-level chip inspection) to capture defect details. Heat source excitation module Laser excitation unit: Single-line / multi-line laser phase-locked thermal imaging: Converts continuous laser (such as 808nm) into a linear beam through a cylindrical lens, and uses a galvanometer scanner to achieve horizontal / vertical scanning to detect random cracks with a sensitivity of 10 2 -110μK, resolution ≤4.8μm, capable of detecting 1.35mm depth delamination defects (430μm micropores) on PCB boards.

[0263] Multi-point pulse laser excitation: uses fixed / adjustable multi-point array laser to simultaneously excite multiple thermal wave points, suitable for large-area detection (such as power transformers, multi-layer chips), with a sensitivity of 110μK and a resolution of ≤4μm, and can detect 1.38mm deep delamination defects (438μm micropores).

[0264] Other heat sources: support pulsed thermal imaging (PT), phase lock-in thermal imaging (LT), ultrasonic vibration thermal imaging (UVT), and eddy current thermal imaging (ECT).

[0265] Control and synchronization unit Excitation unit: includes an arbitrary waveform generator (such as Keysight 33600A), a laser driver (such as Thorlabs), and a synchronization trigger module (PCL) to achieve precise control of laser power, pulse frequency, and scanning path.

[0266] Carrier and adjustment table: Equipped with a product cross adjustment table, fixture and rotation drive system, it supports position calibration and multi-angle detection of inspection objects (such as PCB boards and chips).

[0267] (2) Intelligent algorithm processing platform

[0268] AI algorithm system: Based on the YOLOV5-CDF-Mosaic-MSA algorithm, it integrates feature fusion (CDF), data enhancement (Mosaic), and multi-scale attention mechanism (MSA), and supports defect classification (such as cold solder joints, cracks, and contamination) and positioning (2D / 3D visualization).

[0269] Hardware: A high-performance workstation (such as the Precision series) equipped with a multi-core processor (Intel i9), 64GB+ of RAM, high-speed data interfaces (USB 3.0 / PCI-E), and communication interfaces (RS-232 / Ethernet). Visualization and Storage: Supports real-time display of thermal images, defect annotation (e.g., lq - missing solder ball, xh - cold solder joint), historical data storage, and fault diagnosis report generation.

[0270] (3) Auxiliary hardware unit: electron microscope: used for auxiliary observation of microscopic defects of micron-level multi-layer chips.

[0271] Environmental control module: Optional constant temperature and humidity chamber to reduce the interference of ambient temperature on thermal imaging.

[0272] 2. Workflow

[0273] (1) Preparation before testing

[0274] Object positioning and parameter setting:

[0275] Secure the inspection object (such as a PCB or chip) to the carrier and calibrate its position using the cross-adjustment stage. Select the heat source mode based on the inspection object: Single-line / Multi-line laser lock-in thermal imaging: Suitable for detecting open / short circuits and small cracks (such as welding defects in gas meters).

[0276] Multi-point pulse laser excitation: suitable for internal defect detection of large-area multi-layer structures (such as power transformers and multi-layer chips).

[0277] Set the infrared camera parameters (band, resolution, frame rate), laser power (voltage of a few volts to tens of volts, current accuracy in the milliampere range), and scanning path.

[0278] Image acquisition:

[0279] The excitation unit triggers the laser beam to thermally excite the inspection object, and the high-speed infrared camera synchronously collects thermal response images (including normal / defective states, such as temperature distribution differences of voids and cold solder joints).

[0280] Close-up / electron microscopes perform local high-definition imaging of suspicious areas to obtain microscopic details (such as solder ball morphology and crack direction).

[0281] (2) Data processing and algorithm training

[0282] Image preprocessing:

[0283] The original thermal images are denoised and contrast enhanced, and then divided into training set (70%) and test set (30%) in proportion.

[0284] Manually label the defect type (such as mqxs-missing protrusion, lk-crack), location (such as the upper left corner area defect zjsq) and severity (such as mild / severe aging of the packaging material).

[0285] Algorithm training and verification:

[0286] The annotated images are input into the YOLOV5-CDF-Mosaic-MSA algorithm neural network, and the sample diversity is expanded through Mosaic data enhancement. The CDF module is used to fuse multi-scale features, and the MSA module is used to improve the defect location accuracy.

[0287] After training, the test set is used to verify the model performance. The evaluation indicators include detection accuracy (mAP), recall rate (Recall), positioning error (pixel level), introduction of small target recall rate (Small-Object-Recall), average intersection-well ratio (mloU), area under the precision-recall curve (AUPRC), CIOU LOSS adding small target detection penalty term, Focal Loss, and loss evaluation indicators for weak target detection with large noise: related indicators of the computational mathematical model for evaluating the quality of anchor frames in micron chip defect and fault detection, non-monotonic focusing factor r and outlier β, anchor frame weight setting in the loss function, and the overall performance evaluation indicator is P.

[0288] (3) Real-time detection and fault diagnosis

[0289] Online detection:

[0290] When the object to be tested enters the detection area through the vehicle, the system automatically triggers laser excitation and image acquisition. A single test takes ≤5 seconds (depending on the detection area).

[0291] The thermal image is transmitted to the AI ​​platform in real time, and the algorithm quickly outputs the defect classification results (such as "qlwy-solder overflow, confidence level 95%) and positioning coordinates (such as PCB board coordinates X = 120mm, Y = 80mm).

[0292] Diagnosis and decision making:

[0293] Visual display: 2D / 3D heat maps are used to intuitively display defect location, shape, and temperature gradients, supporting historical data comparison (such as the trend of packaging material aging over time).

[0294] Storage and feedback: The inspection results are stored in the database and a diagnostic report is generated containing the defect type, severity, and recommended measures (such as repair welding and component replacement).

[0295] Process optimization: Combined with defect distribution (such as high frequency of cold solder joints), reverse tracing the production process (such as welding temperature and pressure parameters), optimize process parameters to reduce the defect rate.

[0296] 3. Heat source selection logic is shown in the following table:

[0297] Table 3

[0298]

[0299] 4. Application scenarios and effects Power system: Detect transformer winding overheating, loose cable joints, poor contact of circuit breakers, etc., to prevent power outages caused by thermal defects.

[0300] Electronics manufacturing: Online inspection of chip packaging defects (ball leakage, solder overflow) and PCB multi-layer soldering quality (delamination, micropores) has increased the yield rate to over 99.5%.

[0301] Smart meters: Automated detection of welding defects (such as cold solder joints) in water, electricity, and gas meter sampling components, reducing after-sales failure rates by over 30%.

[0302] Reliability prediction: Through long-term thermal image tracking, the aging cycle of chip packaging materials is predicted, providing early warning of potential failure risks.

[0303] Through the deep integration of hardware high-precision imaging and AI algorithms, this device realizes the full process closed loop of "real-time detection - precise positioning - intelligent diagnosis - process optimization" of power product defects, significantly improving detection efficiency and reliability, and is suitable for intelligent manufacturing and operation and maintenance scenarios in the Industry 4.0 environment.

[0304] Further, the technical principle and functional realization synergy of the invention of the active excitation infrared imaging intelligent detection system are explained as follows:

[0305] The system of the present invention integrates active thermal excitation technology with intelligent algorithms to achieve multi-dimensional detection and full-process control of power product defects. The core principles and functions are as follows:

[0306] 1. Hardware system collaborative working mechanism

[0307] The active excitation infrared imaging system consists of a heat source excitation module, a photoelectric detection module, and a control unit. It uses multiple technologies to capture defect characteristics:

[0308] Heat source excitation module

[0309] Continuous laser excitation (applicable to surface defects):

[0310] The linear laser generator emits a continuous laser beam (e.g. 808nm wavelength) under the control of a control unit (e.g. arbitrary waveform generator, laser driver), which is converted into a linear beam by a cylindrical lens to illuminate the object being measured. When there are defects such as cracks, wear, position offset, etc. on the surface of the object, the reflection and scattering characteristics of the laser at the defect will change, causing abnormal local heat distribution. For example: Crack detection: The laser generates a difference in heat reflection at the edge of the crack, and the thermal infrared camera captures the temperature gradient mutation (sensitivity up to 10 2 μK level), combined with close-up photography (such as electron microscope), it can identify tiny cracks of 4.8μm level.

[0311] Abnormal heating detection: A thermal infrared camera (with a resolution of 20 megapixels or more) scans the surface temperature distribution in real time to locate overheating areas caused by overload or short circuit (such as abnormally high temperature of circuit board components). Close-up shots further capture details of component damage (such as carbonized solder joints).

[0312] Multi-point pulse laser excitation (applicable to internal defects):

[0313] A high-speed thermal infrared camera (frame rate ≥ 200 frames / s) coupled with a fixed / adjustable multi-point array laser emits a pulsed laser beam to stimulate thermal waves on the surface of the inspection object (e.g., multi-layer chips, PCBs). When the thermal wave penetrates the material, if it encounters internal defects (e.g., pores, delamination, or cold solder joints), it will attenuate the thermal wave or shift its phase due to differences in thermal conductivity. For example, in weld quality inspection, a pulsed laser heats the weld area. Cold solder joints or unfused areas experience hindered heat conduction, leading to a delayed local thermal response. The high-speed camera captures the dynamics of the thermal wave diffusion and locates the defect (resolution ≤ 4μm).

[0314] Internal material defect detection: For metal / ceramic materials, multi-point thermal wave excitation can penetrate 1.38mm deep and detect 438μm-level micropores or delamination defects. Combined with close-up shots, the surface hot spot morphology can be observed to determine the defect type (such as looseness and inclusions).

[0315] Photoelectric detection module

[0316] Multi-band infrared imaging:

[0317] The camera supports switching between near-infrared (0.75-1.5μm), mid-infrared (1.5-20μm), and far-infrared (20-1000μm) to meet different inspection needs: Near-infrared enhanced mode: used for chip internal structure perspective (such as ESAM chip packaging defects), with a resolution of 1920×1080;

[0318] Color mode: 4K resolution (3840×2160) for visual analysis of appearance defects in power electronic products (such as identification of surface contaminant types);

[0319] Black and white mode: 20-megapixel resolution (2448×2048) accurately captures temperature details (dynamic range 200dB, pixel depth 16-bit).

[0320] High-precision focus and zoom:

[0321] Based on phase detection autofocus (PDAF) technology and equipped with a 2-120mm zoom lens, it can quickly switch from wide-angle global scanning to telephoto local close-up, suitable for the multi-scale requirements of power transformers (large-area detection) and micron-level chips (sub-micron precision).

[0322] Control and synchronize triggering and motion control with the carrier unit:

[0323] The control unit (e.g., a high-performance workstation) synchronizes laser excitation, camera acquisition, and carrier motion (e.g., product cross adjustment stage, rotary drive system) via an RS-232 / Ethernet interface, ensuring precise matching of thermal image acquisition and the position of the inspection object.

[0324] Environmental adaptability design:

[0325] Supports 0.01lux ultra-low illumination environment (back-illuminated CMOS sensor) and is equipped with a C / F interface high-transmittance optical lens to reduce light loss and adapt to the complex lighting conditions on-site of power equipment.

[0326] 2. Intelligent Algorithm Processing Flow

[0327] The YOLOV5-CDF-Mosaic-MSA algorithm system performs full-process intelligent analysis of thermal images collected by hardware, achieving a closed loop from data to decision-making:

[0328] Image preprocessing and annotation

[0329] The original thermal images were categorized by product type (e.g., PCB, MCU). Mosaic data augmentation technology was used to increase sample diversity. Defect types (e.g., lq - missing solder balls, lk - cracks), locations (e.g., zjsq - upper left corner defect), and severity (e.g., mild / severe packaging material degradation) were manually annotated. The dataset was split into a training set (containing normal / defective images) and a test set in a 7:3 ratio. The dataset was then fed into the YOLOV5 neural network for feature learning.

[0330] Multi-level feature fusion and detection

[0331] CDF (cross-layer feature fusion) module: fuses shallow detail features (such as solder ball morphology) with deep semantic features (such as defect categories) to improve the detection accuracy of small targets (such as 430μm microvias).

[0332] MSA (Multi-Scale Attention Mechanism): Assigns attention weights to images of different resolutions (such as 4K color images and 1080P near-infrared images) to enhance defect recognition in complex backgrounds (such as potential solder joints in multi-layer chip stacking structures).

[0333] Real-time inference: The trained model is deployed to the AI ​​platform to perform real-time analysis of thermal images collected online, outputting defect location coordinates (with pixel-level accuracy), classification results (confidence ≥ 95%), and 2D / 3D thermal visualization maps.

[0334] Diagnostic decision making and process optimization

[0335] Performance evaluation: Evaluate the model detection effect through indicators such as confusion matrix and mAP, and dynamically adjust algorithm parameters (such as learning rate and anchor box size) to ensure continuous optimization of detection accuracy.

[0336] Diagnostic report generation: The storage and processing unit generates a diagnostic report based on defect type (e.g., electrical short (dqdl) or resistance anomaly (dzcy), location, and historical data, including risk level and repair recommendations (e.g., soldering temperature adjustment). This report is then pushed to the production or maintenance end. Table 4 shows the diagnostic report.

[0337] Process closed-loop optimization: Based on defect distribution statistics (such as the occurrence rate of chip placement offset tzpy in a certain batch is greater than 5%), reverse tracing of production process parameters (such as placement head positioning accuracy and solder paste printing thickness).

[0338] Table 4

[0339]

[0340] The technical innovations and advantages are supplemented as follows: Multimodal thermal excitation fusion: the first single-line / multi-line laser phase-locking and multi-point pulse laser collaborative technology, taking into account both high-resolution detection of surface defects and deep penetration of internal defects. Deep optimization of intelligent algorithms: The CDF-Mosaic-MSA module combination based on YOLOV5 breaks through the bottleneck of traditional thermal imaging detection for tiny defects (<5μm) and complex structures (multi-layer stacking). Full-process automated closed loop: end-to-end automation from image acquisition, defect recognition to process optimization, the detection efficiency is more than 80% higher than traditional manual visual inspection, and the false detection rate is <0.5%. Through the deep integration of "hardware precise perception + algorithm intelligent decision-making", the system provides an innovative solution for power product defect detection with high precision, high adaptability and high timeliness, significantly improving the industry's intelligent manufacturing and operation and maintenance level.

[0341] The technical problems solved by the present invention are as follows: The problem of detecting tiny defects and hidden faults: It breaks through the limitations of traditional thermal imaging and R-CNN, YOLO and other algorithms in their ability to detect tiny defects (<5μm) and hidden faults, and solves the problem of failure of early fault warning due to the difficulty in identifying microscopic defects on the surface and sub-surface of products (such as voids inside chips and delamination of PCB boards). The problem of poor adaptability to complex environments: It overcomes the defects of low detection accuracy of existing systems in low signal-to-noise ratio (SNR), low contrast, and strong noise interference environments, and solves the problems of false detection and missed detection caused by poor infrared image quality under complex conditions such as insufficient lighting and strong electromagnetic interference on-site for power equipment. The bottleneck of algorithm performance: It improves the problem of fine information loss and insufficient small target feature extraction caused by strided convolution or pooling layers in traditional intelligent networks, and solves the technical bottleneck of low recall rate and inaccurate positioning of small targets (such as solder balls and protrusions). The contradiction between detection efficiency and real-time performance: It solves the problem that traditional detection methods are time-consuming and cannot meet the real-time detection needs of power product production and operation and maintenance, realizes rapid identification and positioning of defects, and improves detection efficiency. Category imbalance and difficulty in detecting rare defects: Overcome the problem that the defect categories of power products are diverse and unevenly distributed, resulting in the model's weak ability to detect rare defects (such as aging of packaging materials and potential cracks), and improve the model's detection accuracy for all categories of defects.

[0342] The technical effects of the invention are as follows:

[0343] Detection accuracy and reliability are significantly improved

[0344] High-precision defect identification: Through the optimization of the YOLOV5-CDF-Mosaic-SW-MSA algorithm, it can detect micron-level defects (such as 4.8μm cracks and 430μm micropores) with a detection accuracy of 99.96%, an improvement of more than 17% over traditional algorithms.

[0345] Full category coverage: Covers all types of defects, from abnormal surface heating and cracks to internal material defects and electrical performance failures. Indicators such as small target recall rate (SOR) and mean intersection over union (mIOU) are comprehensively optimized, reducing the false detection and missed detection rate to below 0.5%.

[0346] Strong environmental adaptability and robustness

[0347] Multi-Scene Compatibility: Supports multi-band detection across near-infrared, mid-infrared, and far-infrared, adapts to ultra-low illumination environments as low as 0.01 lux, and features a high-transmittance lens and anti-interference hardware to maintain stable performance in complex industrial scenarios. Data Enhancement and Noise Reduction: The Mosaic data enhancement algorithm, combined with adaptive weight distribution, improves the model's ability to handle noisy images and enhances its robustness in varying lighting and electromagnetic environments.

[0348] Breakthrough in detection efficiency and real-time performance

[0349] Fast detection response: The detection speed reaches 29ms / time, and the multi-layer chip detection frame rate is 200 frames / second, meeting the needs of online full inspection of power products and improving efficiency by more than 80% compared with traditional methods.

[0350] Automated closed loop: Automates the entire process of "image acquisition - intelligent analysis - diagnostic decision-making - process optimization", reducing manual intervention and shortening the detection cycle.

[0351] Intelligent diagnosis and decision optimization

[0352] Precise positioning and visualization: Pixel-level defect positioning combined with 2D / 3D heat map visualization intuitively displays defect location, morphology, and temperature gradient, assisting in operation and maintenance decision-making.

[0353] Predictive maintenance: Track defect development trends through historical data, predict component aging cycles and potential failures, and achieve a shift from reactive maintenance to proactive prevention.

[0354] Wide application and industry value

[0355] Multi-field coverage: Applicable to all categories of power electronic products such as power transformers, chip packaging, PCB boards, smart meters, etc., supporting customized testing of products of different sizes and structures.

[0356] Promote industrial upgrading: Help enterprises improve product yield (yield rate in the electronic manufacturing field increased to 99.5%+), reduce operation and maintenance costs, and promote the power electronics industry to upgrade to intelligent and high-precision detection.

[0357] This invention provides a high-precision, highly adaptable intelligent detection and diagnosis system for electronic product defects. Taking into account the characteristics of flip-chip package images, the system determines the appropriate input size based on the actual size of the image captured by the imaging device to highlight small object features. The input module also utilizes the Mosaic data augmentation algorithm to stitch multiple images into a suitable size, enriching the dataset and enhancing model robustness and small object detection capabilities.

[0358] The present invention provides a high-precision and high-adaptability intelligent detection and diagnosis system for electronic product defects. It adopts an active infrared thermal imaging system and a YOLOV5-CDF-Mosaic-SW-MSA image depth processing method to perform intelligent non-destructive detection and diagnosis of surface and sub-surface defects in power and electronic system products. It detects, diagnoses and visually tracks surface and sub-surface defects through deep learning.

[0359] See also Figure 9 , Figure 9 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 9As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 9 A processor 10 is taken as an example.

[0360] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0361] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0362] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0363] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0364] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0365] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0366] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects, characterized by: The method comprises: Acquire image data of chip packages and electrical devices, and establish a training data set based on the image data; Fault detection models suitable for chip packaging and electrical equipment are constructed respectively. In the input module of the fault detection model, the input size of the fault detection model is determined according to the image size, and the mosaic data enhancement algorithm is used to optimize the small target detection capability. The SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability. Multi-scale feature fusion optimization and spatial pyramid pooling adjustment operations are performed in the neck module of the fault detection model. The attention mechanism of the SW-MSA module is adjusted in the prediction module of the fault detection model. Training the fault detection model using the training data set, constructing an evaluation index and calculating a loss function, and adjusting the fault detection model according to the evaluation index and the loss function; The fault detection model is used to perform defect fault detection and diagnosis on power products.

2. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 1 is characterized in that: Acquiring image data of chip packages and electrical devices and establishing a training data set based on the image data includes: Acquire image data of chip packaging and electrical equipment, and perform histogram equalization processing on the image data; Use image annotation tools to annotate image data and establish corresponding annotation types; The labeled image data is used as training data to establish a training dataset.

3. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 2, characterized in that: In the input module of the fault detection model, the input size of the fault detection model is determined according to the image size and the Mosaic data enhancement algorithm is used to optimize the small target detection capability, including: In the input module, the input size of the fault detection model is determined according to the image size; The image selection model based on probability distribution assigns different selection probabilities to each image according to its category label and feature importance, and selects the corresponding category images for splicing in the Mosaic process according to the probability; Based on the affine transformation matrix, the image's rotation angle, scaling ratio, translation position, and center of gravity are automatically determined according to its content features during stitching, so that it can be rotated by a preset angle and translated to a preset position in the Mosaic image. For each adjacent image pair among multiple images participating in Mosaic stitching, the weights of the pixels in their boundary areas are calculated and the pixel values ​​are fused.

4. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 1, characterized in that: The SW-MSA module is introduced into the basic module of the fault detection model to optimize the small target feature extraction capability, including: Dynamically adjust the window size according to the size of the input feature map; The number of windows is dynamically adjusted along the height and width directions according to the step size parameter.

5. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 1, characterized in that: Multi-scale feature fusion optimization and spatial pyramid pooling adjustment operations are performed on the neck module of the fault detection model, including: Increase the number of upsampling and downsampling, and set the size of multiple feature map fusion; Adopt multiple aggregation strategies and assign different weights according to the importance of feature maps of different scales for small target detection; In the spatial pyramid pooling part, set pooling operations of multiple spatial sizes; In the stitching operation after pooling, the mapping method of the number of channels is adjusted according to the feature richness and target number of the image.

6. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 1, characterized in that: Adjusting the attention mechanism of the SW-MSA module in the prediction module of the fault detection model includes: Calculate the target based on the self-attention within the window and adjust the window size; Adjust the number of heads in a multi-head mechanism based on the target feature distribution of chip packages and electrical devices.

7. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 1, characterized in that: The fault detection model is trained using the training data set, an evaluation index is constructed and a loss function is calculated, and the fault detection model is adjusted according to the evaluation index and the loss function, including: During the training of the chip package fault detection model, the training dataset is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the model performance indicators on the validation set under different batch values. Gradually increase the number of training rounds based on the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance indicators meet the preset conditions; Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate; Adopt a learning rate decay strategy and adjust the decay rate and period according to the performance changes of the model on the validation set; Performing a linear transformation on the feature vectors within the window to obtain a query vector, a key vector, and a value vector using an adaptively learned weight matrix, wherein the weight matrix is ​​adaptively adjusted during training according to the back propagation of the loss function; For each head, a learnable bias matrix is ​​introduced to perform multi-head attention calculation in each window to obtain the attention score; Perform weighted summation of the attention score and value vector to obtain the single-head output feature vector; The output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

8. The high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to claim 7, characterized in that: The fault detection model is trained using the training data set, an evaluation index is constructed and a loss function is calculated, and the fault detection model is adjusted according to the evaluation index and the loss function, further comprising: In the process of training the fault detection model for electrical equipment, the training data set is randomly divided into several batches. The loss is calculated and the model parameters are updated on each batch. The optimal batch size is determined by comparing the changes in the performance indicators of the model on the validation set under different batch values. Gradually increase the number of training rounds based on the model convergence and performance improvement trend, and stop increasing the number of training rounds when the performance indicators meet the preset conditions; Set the initial learning rate, observe the model convergence under different initial learning rates based on small-scale data set experiments, and determine the optimal value of the initial learning rate; A small target detection penalty term is added to CIOU_LOSS. The weight coefficient is determined according to the ratio of target size to image size, which increases the weight of small target position and category error. At the same time, Focal Loss is used to adjust the adjustable parameters to solve the category imbalance problem. Introducing multiple evaluation indicators to assess the performance of the model in flip-chip electrical equipment detection in order to adjust and optimize the model; For each head, a learnable bias matrix is ​​introduced to perform multi-head attention calculation in each window to obtain the attention score; Perform weighted summation of the attention score and value vector to obtain the single-head output feature vector; The output feature maps of all heads are concatenated along the channel dimension to obtain the final output feature map.

9. A high-precision and high-adaptability intelligent detection and diagnosis system for electronic product defects, characterized by: The system includes: an active excitation infrared imaging zoom camera, an industrial-grade CMOS sensor, an excitation unit, a laser driver, a CW laser, a reflector, a programmable logic controller, a cylindrical lens, a beam exciter, a heat spreader, An active-excitation infrared imaging zoom camera captures thermal images of chip packaging and electrical equipment, and divides the thermal images into a training learning image group and a test detection image group in proportion. Images of defective faults and non-defective faults in the two groups are added to the YOLOV5 algorithm system respectively. The thermal images are classified by product, and then the classified images are used to implement the training learning set and detection test set of the YOLOV5 algorithm respectively. After the quality of the training learning set and the detection image meets the performance evaluation indicators, the images are inspected to identify the fault defect location and classification.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the high-precision and high-adaptability intelligent detection and diagnosis method for electronic product defects according to any one of claims 1 to 8.

Images