Electronic component solder joint fault diagnosis method and system based on image recognition
By preprocessing and perturbation analysis of solder joint images, key pixel clusters are identified and visualized, solving the problem of opacity in the solder joint fault diagnosis process and improving the credibility of diagnostic conclusions and the efficiency of verification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG PUYU ELECTRONIC TECH CO LTD
- Filing Date
- 2026-05-26
- Publication Date
- 2026-06-23
Smart Images
Figure CN122265746A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more specifically, to a method and system for diagnosing solder joint faults in electronic components based on image recognition. Background Technology
[0002] In the manufacturing process of electronic products, automated optical inspection of solder joints on circuit boards is a crucial step in ensuring product quality. Conventional image recognition-based diagnostic methods typically acquire solder joint images, extract their geometric or textural features, and compare them with standard models to determine the presence of faults such as cold solder joints or insufficient solder. However, on actual production lines, due to variations in lighting at the camera station, slight lens contamination, or minor deviations in circuit board positioning, the quality of the acquired solder joint images often fluctuates, exhibiting characteristics such as localized overexposure, insufficient brightness, or slight blurring. This fluctuation in image quality directly interferes with the accuracy of feature extraction, rendering the diagnostic algorithm's judgment unreliable. A core challenge of existing technologies lies in the fact that their diagnostic process is like a black box; the system only outputs a pass or fail conclusion, but when the conclusion is fail, it cannot clearly identify which specific area or appearance feature in the image caused this judgment. This poses a significant challenge to manual review, requiring reviewers to painstakingly search for potential defects in poor-quality images. The diagnostic conclusion lacks direct visual evidence, resulting in low review efficiency and subjective results, severely impacting the effectiveness of fault tracing and process improvement.
[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this application provides a method and system for diagnosing solder joint faults in electronic components based on image recognition. This method and system can solve the technical problems in existing technologies, such as the lack of transparency in the solder joint fault diagnosis process, the lack of direct visual evidence for the diagnostic conclusions leading to difficulties in verification, and low reliability.
[0005] In a first aspect, this application provides a method for diagnosing solder joint faults in electronic components based on image recognition, comprising: Acquire solder joint images of the electronic components to be diagnosed, and preprocess the solder joint images; In the preprocessed solder joint image, the solder joint area is located and the image features of the solder joint area are extracted. Based on the extracted image features, the solder joint fault type is preliminarily determined and the reliability value of the preliminary determination result is calculated. The solder joint area is divided into multiple pixel sub-regions, and a perturbation operation is performed on each pixel sub-region in turn. The perturbation operation is used to simulate the changes in image information within the pixel sub-region. After each perturbation operation, the solder joint fault type is determined a second time based on the image features of the solder joint image after the perturbation operation, and the reliability value of the second determination result is calculated. The difference between the reliability value of the initial determination result and the reliability value of the second determination result is used as the contribution value of the pixel sub-region. Based on the magnitude of the contribution value, at least one key pixel cluster is identified from all pixel sub-regions, and the key pixel cluster is visually marked on the solder joint image as visual evidence for fault diagnosis.
[0006] This technical solution not only diagnoses solder joint faults, but more importantly, through disturbance analysis and contribution calculation, it can accurately locate and visually mark the key image areas that lead to fault determination, providing direct visual evidence for the diagnostic results. It solves the technical problems of opaque diagnostic process and difficulty in verifying results in traditional methods, and significantly improves the credibility and interpretability of diagnostic conclusions.
[0007] Furthermore, the preprocessing steps for the solder joint images include: Calculate the global average brightness value of the solder joint image, and perform brightness correction on the solder joint image based on the comparison result between the global average brightness value and the preset target brightness value; Median filtering is used to remove noise from the solder joint image, and the brightness value of each pixel in the solder joint image is replaced with the median of the brightness values of its neighboring preset pixel areas.
[0008] This technical solution performs brightness correction and median filtering on the image before diagnosis begins, which can effectively eliminate image quality fluctuations caused by uneven lighting or camera noise. This provides a stable and clear image foundation for accurate solder joint location and feature extraction, thereby improving the anti-interference capability and stability of the entire diagnostic method.
[0009] Furthermore, the step of locating the solder joint region in the preprocessed solder joint image includes: Obtain the preset standard solder joint grayscale template; In the preprocessed solder joint image, the normalized cross-correlation coefficient between each window region and the preset standard solder joint grayscale template is calculated using a preset sliding window. Regions with normalized cross-correlation coefficients higher than a preset threshold are identified as candidate regions, and the region with the highest normalized cross-correlation coefficient among the candidate regions is selected as the solder joint region.
[0010] This technical solution employs a normalized cross-correlation matching method based on standard solder joint templates, which can accurately and stably locate solder joint areas from complex circuit board backgrounds. This effectively avoids positioning errors caused by fixture positioning offsets or interference from adjacent components, ensuring that subsequent feature extraction and analysis are always performed on the correct target area.
[0011] Furthermore, the image features include geometric shape features characterizing the geometry of the solder joint, texture features characterizing the microstructure of the solder joint surface, and brightness distribution features characterizing the brightness distribution of the solder joint.
[0012] This technical solution utilizes three types of features—geometric shape, texture, and brightness distribution—to comprehensively describe the solder joint status from multiple dimensions, including macroscopic morphology, surface microstructure, and solder uniformity. Compared to a single feature, it can more accurately capture subtle differences between different types of faults, improving the accuracy of preliminary judgment.
[0013] Furthermore, the steps of making a preliminary determination of the solder joint fault type based on the extracted image features and calculating the reliability value of the preliminary determination result include: According to the preset fault determination rules, the geometric shape features, texture features, and brightness distribution features are compared with the corresponding preset determination thresholds to determine the solder joint fault type. The system calculates the degree of deviation between geometric features, texture features, and brightness distribution features and the corresponding preset judgment thresholds, and generates a reliability value by weighting the deviations.
[0014] This technical solution achieves fault determination based on explicit rules by comparing extracted multi-dimensional features with preset thresholds, avoiding subjective judgment. Simultaneously, by calculating the degree of feature deviation and generating a weighted reliability value, a quantitative confidence assessment is provided for the preliminary determination results, offering a benchmark reference for subsequent disturbance analysis.
[0015] Furthermore, the steps of performing the perturbation operation on each pixel sub-region sequentially include: Replace the brightness value of all pixels within a pixel sub-region with the average brightness value of the surrounding pixels of the pixel sub-region.
[0016] This technical solution replaces the sub-region with the average brightness of surrounding pixels, offering a computationally simple and efficient perturbation method. This operation simulates the blurring or loss of local image information, enabling rapid assessment of the region's impact on overall feature stability, making it suitable for applications with high computational efficiency requirements.
[0017] Furthermore, the steps of performing the perturbation operation on each pixel sub-region sequentially include: Call the preset background reference library containing a clean and flawless background texture; Extract the surrounding environment texture of the pixel sub-regions sequentially, and retrieve matching reference textures from the background reference library; Replace the image information of the pixel sub-region with the retrieved reference texture.
[0018] This technical solution uses a clean texture retrieved from a background reference library for replacement, which more realistically simulates the state after a suspected defective area has been repaired to a normal background. This perturbation method is particularly effective in identifying minute defects superimposed on normal textures, and its analysis results are more physically meaningful, improving the accuracy of contribution calculation.
[0019] Furthermore, the step of identifying at least one key pixel cluster from all pixel sub-regions based on the magnitude of the contribution value includes: Sort all pixel sub-regions in descending order of their contribution value; The pixel sub-regions whose contribution values are ranked first in the first preset number of positions are determined as the first key pixel cluster; The pixel sub-regions whose contribution values are ranked after the first preset number of positions and the second preset number of positions are determined as the second key pixel cluster.
[0020] This technical solution, by sorting and hierarchically selecting contribution values, not only identifies the core area (first key pixel cluster) that plays a decisive role in fault determination, but also identifies the related areas (second key pixel cluster) that play an auxiliary or secondary role. This achieves hierarchical analysis of fault-related areas, making the presentation of visual evidence more logical and in-depth.
[0021] Furthermore, the step of visually marking key pixel clusters on the solder joint image as visual evidence for fault diagnosis includes: The first key pixel cluster is highlighted with the first color, and the second key pixel cluster is highlighted with the second color.
[0022] This technical solution uses different colors to distinguish and mark the first and second key pixel clusters, enabling reviewers to visually and clearly differentiate the core cause area and secondary influence area of the fault. This greatly improves the readability and information transmission efficiency of the diagnostic report, facilitating rapid problem location and process traceability.
[0023] Secondly, this application also discloses an image recognition-based electronic component solder joint fault diagnosis system for performing any of the foregoing image recognition-based electronic component solder joint fault diagnosis methods. The system includes: The image acquisition and preprocessing module acquires solder joint images of the electronic components to be diagnosed and preprocesses the solder joint images. The preliminary judgment module locates the solder joint area in the preprocessed solder joint image, extracts the image features of the solder joint area, makes a preliminary judgment on the solder joint fault type based on the extracted image features, and calculates the reliability value of the preliminary judgment result. The perturbation analysis module divides the solder joint area into multiple pixel sub-regions and performs perturbation operations on each pixel sub-region in turn. The perturbation operations are used to simulate changes in image information within the pixel sub-region. The secondary judgment module performs a secondary judgment on the solder joint fault type based on the image features of the solder joint image after each disturbance operation, and calculates the reliability value of the secondary judgment result. The difference between the reliability value of the preliminary judgment result and the reliability value of the secondary judgment result is used as the contribution value of the pixel sub-region. The key pixel cluster identification module identifies at least one key pixel cluster from all pixel sub-regions based on the magnitude of the contribution value, and visually marks the key pixel cluster on the solder joint image as visual evidence for fault diagnosis.
[0024] The technical solution provided in this application fundamentally solves the problems of opaque diagnostic processes and lack of interpretability in traditional solder joint fault diagnosis methods by introducing an innovative perturbation analysis mechanism. This method quantifies the contribution of each pixel region to the final fault determination by systematically changing local areas of the solder joint image and observing the corresponding changes in the diagnostic results. Based on this contribution, the system can automatically identify and highlight key pixel clusters that lead to the fault determination. This transforms abstract diagnostic conclusions into intuitive visual evidence, allowing for immediate identification of problems during manual review, greatly improving review efficiency and accuracy. Furthermore, by focusing on high-contribution key areas, the reliability of the diagnosis is no longer easily affected by irrelevant noise or background textures in the image, thus enhancing the method's robustness to image quality fluctuations such as changes in lighting and slight blurring. Ultimately, this application not only provides a more reliable diagnostic tool but also establishes a trustworthy and traceable diagnostic and review process. Attached Figure Description
[0025] Figure 1 This is a flowchart illustrating the image recognition-based electronic component solder joint fault diagnosis method provided in an embodiment of this application.
[0026] Figure 2 This is a schematic diagram of the structure of an image recognition-based electronic component solder joint fault diagnosis system provided in an embodiment of this application.
[0027] Labeling Explanation: 210, Image Acquisition and Preprocessing Module; 220, Preliminary Judgment Module; 230, Disturbance Analysis Module; 240, Secondary Judgment Module; 250, Key Pixel Cluster Recognition Module. Detailed Implementation
[0028] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0030] In the automated production and assembly processes of electronic components, inspecting the quality of solder joints is a crucial step in ensuring the reliability of the final product. Especially on high-density packaged circuit boards, where solder joints are tiny and numerous, manual visual inspection can no longer meet the requirements of production efficiency and consistency. Therefore, automated optical inspection systems are widely used. However, a common technical challenge is that when the system determines that a solder joint is faulty, its output is often only an abstract conclusion, such as a cold solder joint or insufficient solder, without providing direct visual evidence leading to this determination. For inspectors on the production line, they are faced with a raw image that may be of poor quality due to uneven lighting or slight vibrations, and an isolated diagnostic conclusion. In this situation, inspectors struggle to quickly and accurately locate the specific defective area that triggered the determination on the image. The diagnostic process becomes like an opaque black box, which not only reduces the efficiency of inspection but also makes fault tracing and targeted process improvements extremely difficult.
[0031] To solve the aforementioned technical problems, firstly, see [reference needed] Figure 1 This application provides a method for diagnosing solder joint faults in electronic components based on image recognition. The method includes: S1. Obtain solder joint images of the electronic components to be diagnosed, and preprocess the solder joint images; S2. Locate the solder joint area in the preprocessed solder joint image and extract the image features of the solder joint area. Based on the extracted image features, make a preliminary judgment on the solder joint fault type and calculate the reliability value of the preliminary judgment result. S3. Divide the solder joint area into multiple pixel sub-regions, and perform a perturbation operation on each pixel sub-region in turn. The perturbation operation is used to simulate the changes in image information within the pixel sub-region. S4. After each disturbance operation, based on the image features of the solder joint image after the disturbance operation, a secondary determination of the solder joint fault type is made, and the reliability value of the secondary determination result is calculated. The difference between the reliability value of the preliminary determination result and the reliability value of the secondary determination result is used as the contribution value of the pixel sub-region. S5. Based on the magnitude of the contribution value, identify at least one key pixel cluster from all pixel sub-regions, and visually mark the key pixel cluster on the solder joint image as visual evidence for fault diagnosis.
[0032] The reliability value is a quantitative representation of the confidence level in a fault determination result. This value is not a simple yes or no judgment, but a continuous value calculated by considering the deviations of various image features from a standard threshold. For example, a value between 0 and 1 indicates that the higher the value, the more the image features match the typical behavior of a certain fault, and the stronger the system's confidence in the determination result.
[0033] Perturbation refers to the artificial and controlled modification of a small local area in an image. This modification aims to simulate the state after the information in that local area disappears or is repaired, in order to observe how much this change will affect the diagnostic results of the entire solder joint image.
[0034] The contribution value refers to the quantitative indicator of the influence of image information from a certain local area on the final fault determination result. If the reliability value of the fault determination drops sharply after perturbing a small area, it indicates that the original information of this small area is a key factor in the system's fault determination, and its contribution value is high. Conversely, if the result remains basically unchanged after perturbation, it indicates that the area is insignificant, and its contribution value is low.
[0035] Key pixel clusters refer to a set of one or more pixel sub-regions with the highest contribution values. These regions are the parts of the image that play a decisive role in fault diagnosis and are the crux of the fault.
[0036] Visual evidence refers to the process of highlighting, outlining, or otherwise making a point of marking key pixel clusters identified on the original solder joint image, thereby directly linking abstract diagnostic conclusions with specific physical locations on the image, forming evidence that can be directly observed and confirmed by the human eye.
[0037] The technical solution provided in this application will be described in detail below.
[0038] In a specific implementation scenario, suppose a surface mount technology (SMT) production line is manufacturing a batch of circuit boards containing ball grid array (BGA) packaged chips. An automated optical inspection (OIO) system deployed on this line captures an image of each solder ball individually using an industrial camera. First, the system acquires an image of the solder joint of the solder ball to be diagnosed; this image might be an 8-bit grayscale image with a resolution of 512 x 512 pixels. Due to fluctuations in lighting conditions or thermal noise from the camera sensor, the original image may contain uneven brightness or random salt-and-pepper noise. Therefore, image preprocessing is necessary. A basic preprocessing approach could be to apply a simple thresholding operation to initially segment the foreground and background, or to use a basic Gaussian filter to smooth the image and reduce the effects of minor noise.
[0039] After preprocessing, the system needs to accurately locate the core region where the solder joint is located within the entire image. One feasible localization method is to utilize the prior knowledge that solder joints typically appear as approximately circular bright spots, and use algorithms such as the Hough circle transform to detect circular targets in the image. Among the multiple detected circular candidate targets, the one whose size best matches the preset solder ball diameter range and is located in the center of the image can be selected as the final solder joint region. For example, the system will use the detected circle's center as the center to extract a 120 by 120 pixel square region as the solder joint region for subsequent analysis.
[0040] Next, the system needs to extract image features that characterize the state of the 120x120 pixel solder joint area. These features are the basis for subsequent fault determination. Based on the extracted image features, the system will make a preliminary judgment on the type of solder joint fault. For example, the system has preset multiple fault types, such as normal, cold solder joint, insufficient solder, and solder bridging. By comparing the extracted feature values with a preset rule base, the system will give a preliminary judgment conclusion. For example, if the calculated solder joint area is much smaller than the standard value, the system may initially judge it as insufficient solder. At the same time, the system will also calculate the reliability value of this preliminary judgment result. This calculation process can be to quantify the degree to which each feature deviates from its normal range and then perform a weighted summation. Suppose the system finally judges it as a cold solder joint and calculates a reliability value of 0.85. This 0.85 is the benchmark for subsequent analysis.
[0041] Next, we move to the core of this application's solution: using perturbation analysis to determine which region caused the cold solder joint. The system divides this 120x120 pixel solder joint area into multiple smaller pixel sub-regions. For example, it can be divided into 144 10x10 pixel sub-regions. Then, the system begins an iterative process, performing a perturbation operation on each sub-region sequentially. Taking the first 10x10 sub-region as an example, a simple perturbation operation could be to set the grayscale value of all 100 pixels within it to a fixed neutral grayscale value, such as 128. This operation is equivalent to temporarily erasing the original image information of this area.
[0042] After perturbating the first sub-region, the modified solder joint image is sent back to the feature extraction and fault determination process. The system recalculates the image features based on this incomplete image and performs a second determination. Assuming the second determination still results in a faulty solder joint, but the calculated reliability value drops to 0.82, the difference between the initial and second determination reliability values (0.85 minus 0.82) equals 0.03. This 0.03 is recorded as the contribution value of the first sub-region. Subsequently, the system restores the original image of the first sub-region and performs the same perturbation operation on the second 10x10 sub-region, calculating its contribution value. This process continues until the contribution values of all 144 sub-regions have been calculated.
[0043] Once the contribution values of all sub-regions have been calculated, the system obtains a contribution matrix. This matrix reflects the distribution of influence of each local area within the solder joint region on the determination of a cold solder joint. For example, it may be found that several adjacent sub-regions have particularly high contribution values, such as 0.2 or even 0.3, while the contribution values of most other regions are close to zero. This indicates that it is precisely these high-contribution regions, whose original image information (e.g., abnormal texture or grayscale) is the main reason why the system determines that it is a cold solder joint. Based on the magnitude of the contribution value, the system identifies several regions with the highest contribution from all 144 pixel sub-regions and combines them into one or more key pixel clusters.
[0044] The final step involves the system visually marking identified key pixel clusters on the original, undisturbed image of the solder joint. For example, a semi-transparent red mask can be used to overlay these high-contribution areas. This way, when the marked image is presented to reviewers along with the diagnostic conclusion of a faulty solder joint, the reviewers no longer need to blindly search the entire image; they can immediately focus their attention on the red-marked areas. They can see that these areas may contain tiny cracks or uneven reflections, which are typical visual signs of a faulty solder joint. In this way, the diagnostic conclusion receives strong visual evidence.
[0045] The above method transforms abstract and inexplicable diagnostic conclusions into visual evidence associated with specific locations in images, greatly improving the credibility and verifiability of diagnostic results and providing precise guidance for subsequent process improvements.
[0046] In a preferred embodiment, the step of preprocessing the solder joint image includes: Calculate the global average brightness value of the solder joint image, and perform brightness correction on the solder joint image based on the comparison result between the global average brightness value and the preset target brightness value; Median filtering is used to remove noise from the solder joint image, and the brightness value of each pixel in the solder joint image is replaced with the median of the brightness values of its neighboring preset pixel areas.
[0047] The reason for this preprocessing is that in actual production, the lighting conditions for shooting different batches and at different times are difficult to guarantee to be completely consistent. An overall dark image may cause a normal solder joint to be misjudged as having insufficient solder, while an overexposed image may mask minute surface cracks. The brightness correction step proposed in this application effectively addresses this problem. Specifically, the system first calculates the average brightness of all pixels in the entire input image, for example, obtaining a value of 90. The system internally presets an ideal target brightness value, such as 128. The system calculates a correction factor, for example, by dividing the target brightness value by the actual average brightness value, resulting in 128 divided by 90, approximately equal to 1.42. Then, the brightness value of each pixel in the image is multiplied by this correction factor of 1.42, and the result is limited to the range of 0 to 255. After this processing, regardless of whether the original image is too dark or too bright, its overall brightness level is unified to a standardized benchmark, thereby eliminating the interference of lighting variations on subsequent feature extraction.
[0048] Furthermore, when camera sensors operate at high temperatures or high gain, isolated bright or dark spots, known as salt-and-pepper noise, can easily appear in images. This noise can be misidentified as tiny solder balls or pinhole defects. Median filtering is a very effective noise reduction method. In practice, the system defines a small neighborhood window, such as a 3x3 pixel square area. For each pixel in the image, the system reads the brightness values of the nine pixels within its 3x3 neighborhood, sorts these nine values, and replaces the original value of the current center pixel with the median value (the fifth largest value). Because noise points are usually isolated extremely bright or dark spots, they will be at opposite ends after sorting, while the median value better represents the true brightness level of the area. In this way, isolated noise points can be effectively filtered out while preserving the edge contour information of solder joints, resulting in a cleaner image.
[0049] In a preferred embodiment, the step of locating the solder joint region in the preprocessed solder joint image includes: Obtain the preset standard solder joint grayscale template; In the preprocessed solder joint image, the normalized cross-correlation coefficient between each window region and the preset standard solder joint grayscale template is calculated using a preset sliding window. Regions with normalized cross-correlation coefficients higher than a preset threshold are identified as candidate regions, and the region with the highest normalized cross-correlation coefficient among the candidate regions is selected as the solder joint region.
[0050] The aforementioned methods for locating solder joints using geometry-based methods such as Hough circle transform, while effective under ideal conditions, suffer from reduced accuracy and robustness when solder joints are irregularly shaped due to poor soldering or when other circular obstructions are present on the circuit board. In contrast, the template-matching method proposed in this application offers greater reliability. Specifically, before deployment, the system pre-collects a large number of well-shaped, centered standard solder joint images and processes them through averaging and smoothing to generate one or more standard solder joint grayscale templates. This template can be considered an ideal photograph of the solder joint, such as a 64x64 pixel image.
[0051] During diagnosis, the system uses a sliding window of the same size as the template (64 x 64 pixels) on the preprocessed image to be tested. This window starts from the top left corner of the image and moves pixel by pixel to the right and down, covering the entire image. At each position, the system calculates the normalized cross-correlation coefficient between the image content within the current window and the standard solder joint template. This coefficient is a value between -1 and 1; the closer the value is to 1, the higher the similarity between the image within the window and the template. Calculating the normalized cross-correlation coefficient effectively resists linear changes in brightness; even if the area used for matching is brighter or darker than the overall template, a high coefficient value can still be obtained as long as the grayscale distribution is similar.
[0052] After calculation, the system generates a correlation coefficient graph of the same size as the original image. The value of each point on the graph represents the degree of matching between the window with that point as its top-left corner and the template. The system sets a preset threshold, such as 0.8, and identifies all regions with coefficient values higher than 0.8 as candidate regions. In general, the actual solder joint region will produce a very prominent peak. Therefore, the system selects the region with the highest normalized cross-correlation coefficient from all candidate regions as the final identified solder joint region. This method utilizes the overall grayscale distribution information of the solder joint, not just its outline shape, thus resulting in more accurate positioning and better resistance to interference.
[0053] In a preferred embodiment, the image features include geometric shape features characterizing the geometry of the solder joint, texture features characterizing the microstructure of the solder joint surface, and brightness distribution features characterizing the brightness distribution of the solder joint.
[0054] Using only a single type of feature often fails to comprehensively describe the complex state of solder joints. For example, looking only at the area size may not distinguish between a rough solder joint with a normal area and a smooth solder joint with a normal area. Therefore, this application proposes to extract features from multiple dimensions to construct a more comprehensive description of solder joint states.
[0055] Specifically, geometric features are used to describe the macroscopic shape of the solder joint. Calculated features include: the total number of pixels in the solder joint area, i.e., the area, which is directly related to the amount of solder; the perimeter of the solder joint area; the roundness calculated from the area and perimeter—a perfectly round solder joint has a roundness close to 1, while solder joints with spikes or bridging will have significantly reduced roundness; and the aspect ratio of the minimum bounding rectangle of the solder joint area, which can be used to detect bridging faults because bridging causes the solder joint to be elongated in a certain direction.
[0056] Texture features are used to describe the microstructure of the solder joint surface, which is crucial for determining the presence of defects such as cold solder joints and cracks. A common texture analysis method is to calculate the gray-level co-occurrence matrix and extract a series of features based on this matrix. For example, contrast can be calculated; a high contrast value usually indicates the presence of areas with abrupt changes in brightness on the solder joint surface, which may correspond to cracks or pits. Energy or angular second moment can be calculated; this value reflects the uniformity of the image's gray-level distribution and the coarseness of the texture; a smooth solder joint surface has a higher energy value. Entropy can also be calculated; a high entropy value indicates a more complex and irregular texture on the solder joint surface.
[0057] Brightness distribution characteristics are used to describe the distribution of solder within the solder joint area. This can be achieved by analyzing the grayscale histogram of the solder joint area. Calculated features include: average brightness, reflecting the overall brightness of the solder joint; brightness standard deviation, a high standard deviation indicates uneven brightness distribution on the solder joint surface, potentially with excessively bright highlights and excessively dark shadows, a common sign of poor soldering; and histogram skewness and kurtosis, these higher-order statistics can more precisely describe the asymmetry and concentration of the brightness distribution.
[0058] By combining these three types of features into a high-dimensional feature vector, the system can comprehensively examine the weld joint from three levels: macroscopic morphology, microscopic structure, and material distribution, laying a solid foundation for subsequent accurate judgment.
[0059] In a preferred embodiment, the steps of making a preliminary determination of the solder joint fault type based on the extracted image features and calculating the reliability value of the preliminary determination result include: According to the preset fault determination rules, the geometric shape features, texture features, and brightness distribution features are compared with the corresponding preset determination thresholds to determine the solder joint fault type. The system calculates the degree of deviation between geometric features, texture features, and brightness distribution features and the corresponding preset judgment thresholds, and generates a reliability value by weighting the deviations.
[0060] The process involves comparing geometric features, texture features, and brightness distribution features with corresponding preset thresholds based on predefined fault determination rules to identify the type of solder joint fault. Specifically, this can be achieved as follows: First, for geometric features such as the diameter, height, and area of the solder joint, a series of threshold ranges can be set. For example, a diameter smaller than a certain value may indicate insufficient solder, while a diameter larger than a certain value may indicate excessive solder. Second, for texture features such as the roughness and uniformity of the solder joint surface, a threshold for the difference between the texture feature value and the standard texture feature value can be set. A large difference may indicate surface oxidation or cracks. Finally, for brightness distribution features such as the average brightness and brightness uniformity of the solder joint area, a threshold for the deviation of the brightness value from the standard brightness value can be set. Excessively high or low brightness may indicate a cold solder joint or a poorly soldered joint. By comparing the extracted geometric features, texture features, and brightness distribution features of the solder joint image with these preset thresholds, the type of fault the solder joint belongs to can be preliminarily determined.
[0061] As a preferred embodiment, the solution of this application is specifically implemented as follows: In a solder joint fault diagnosis system, the first step is to acquire an image of the solder joint to be diagnosed. After preprocessing, the solder joint region is located. Geometric features (such as solder joint diameter and area), texture features (such as surface roughness and uniformity), and brightness distribution features (such as average brightness and brightness gradient) are extracted from this solder joint region.
[0062] Next, according to preset fault determination rules, the extracted geometric features are compared with preset geometric thresholds (e.g., diameter range [1.5mm, 2.0mm]). Simultaneously, texture features are compared with preset texture thresholds (e.g., roughness index less than 0.3), and brightness distribution features are compared with preset brightness thresholds (e.g., average brightness value between 180-220). If the solder joint diameter is 1.2mm, it is initially determined to be a solder shortage fault. Subsequently, the deviation of each feature from its corresponding threshold is calculated. For example, a solder joint diameter of 1.2mm deviates from the lower threshold of 1.5mm by 0.3mm. A texture feature index of 0.4 deviates from the threshold of 0.3 by 0.1. An average brightness value of 150 deviates from the lower threshold of 180 by 30. Finally, based on preset weights, e.g., geometric feature weight 0.5, texture feature weight 0.3, and brightness distribution feature weight 0.2, these deviations are weighted and calculated. A reliability value for the preliminary determination result is generated. For example, the reliability value = 0.5*(0.3 / 1.5) + 0.3*(0.1 / 0.3) + 0.2*(30 / 180) = 0.1 + 0.1 + 0.033 = 0.233. This value represents the reliability of the preliminary judgment result; the lower the value, the worse the reliability and the greater the possibility of failure.
[0063] Through the above technical solution, this application solves the problem of how to preliminarily determine the type of solder joint fault based on extracted image features and calculate the reliability value of the preliminary determination result. By comparing geometric features, texture features, and brightness distribution features with preset judgment thresholds, the type of solder joint fault can be preliminarily determined. By calculating the deviation of these features from the thresholds and performing weighted calculations, a quantified reliability value can be generated, thereby providing more valuable confidence information for fault diagnosis and improving the credibility and practicality of the diagnostic results.
[0064] After determining the preliminary judgment result and its reliability value, in order to investigate the specific image region that led to the judgment, this application further provides a specific method for performing a perturbation operation on the pixel sub-region. In some embodiments, the perturbation operation can be implemented using different strategies.
[0065] An alternative implementation involves performing a perturbation operation on each pixel sub-region sequentially, including: Replace the brightness value of all pixels within a pixel sub-region with the average brightness value of the surrounding pixels of the pixel sub-region.
[0066] This is a computationally simple and efficient perturbation method. In its implementation, taking the aforementioned 10x10 pixel sub-region as an example, the system first defines its surrounding neighboring pixels. A feasible definition is a ring of pixels surrounding the 10x10 sub-region, i.e., a border with a width of 1 pixel. This border contains (10+2) x (10+2) minus 10 x 10, a total of 44 pixels. The system calculates the arithmetic mean of the brightness values of these 44 neighboring pixels. Assume the calculated average is 145. Subsequently, the system uniformly sets the brightness values of all 100 pixels within the 10x10 sub-region to 145. Visually, this operation is equivalent to a strong blurring of the local area, effectively erasing the original texture and brightness details that might contain defect information, thus allowing the system to assess how the diagnostic results would change without information from this area.
[0067] As another, more refined implementation, the step of sequentially performing a perturbation operation on each pixel sub-region includes: Call the preset background reference library containing a clean and flawless background texture; Extract the surrounding environment texture of the pixel sub-regions sequentially, and retrieve matching reference textures from the background reference library; Replace the image information of the pixel sub-region with the retrieved reference texture.
[0068] This method aims to more realistically simulate what a potentially defective area would look like after being repaired to a normal state, rather than simply blurring it. Its implementation is more complex, specifically including: First, a background reference library needs to be built in advance. This library can be built by collecting hundreds of standard solder joint images that have been confirmed by experts as perfect or defect-free. From the smooth, uniformly textured areas of these standard solder joint images, a large number of reference texture blocks, such as 12 by 12 pixels, can be cropped and stored to form the reference library.
[0069] Secondly, when perturbing a 10 by 10 pixel sub-region, the system first extracts the texture of its surrounding environment, that is, the ring of pixels surrounding the sub-region (for example, the border of a 12 by 12 pixel region).
[0070] Then, the system takes this surrounding environment texture and searches the background reference library to find the best matching reference texture. The matching criterion can be to calculate the similarity between two texture blocks, such as using the sum of squared differences (SSD) algorithm. The reference texture block with the smallest sum of squared differences is considered the best match.
[0071] Finally, the system replaces the currently perturbed 10x10 sub-region with the central portion of the retrieved best-matching 12x12 reference texture block (i.e., a 10x10 region). This texture transfer method generates a seamless, natural-looking repair effect that blends into the surrounding environment, making subsequent secondary assessments and contribution calculations more accurate, especially for identifying defects such as tiny cracks or blemishes superimposed on a normal background.
[0072] After calculating the contribution values of all pixel sub-regions, to more clearly present the cause of the fault, this application further provides a method for identifying and classifying key pixel clusters. The step of identifying at least one key pixel cluster from all pixel sub-regions based on the magnitude of the contribution value includes: Sort all pixel sub-regions in descending order of their contribution value; The pixel sub-regions whose contribution values are ranked first in the first preset number of positions are determined as the first key pixel cluster; The pixel sub-regions whose contribution values are ranked after the first preset number of positions and the second preset number of positions are determined as the second key pixel cluster.
[0073] Specifically, assuming the solder joint area is divided into 144 sub-regions, the system has calculated the contribution value of each of these 144 regions. The system will then sort the list of these 144 values in descending order. Subsequently, according to preset parameters, the system will select key regions at different levels. For example, the preset first digit can be set to 5, meaning the top 5 sub-regions in terms of contribution are identified as the first key pixel cluster. These regions are the most core and direct factors leading to fault determination. Simultaneously, the preset second digit can be set to 10, meaning the 10 sub-regions ranking 6th to 15th in contribution are identified as the second key pixel cluster. These regions may represent minor defects, edge extensions of defects, or accompanying phenomena related to the core defect. This hierarchical approach provides richer hierarchical information for fault analysis, distinguishing the primary and secondary issues.
[0074] Finally, in order to present the analysis results to users in the most intuitive way, the steps of visually marking key pixel clusters on the solder joint image as visual evidence for fault diagnosis include: The first key pixel cluster is highlighted with the first color, and the second key pixel cluster is highlighted with the second color.
[0075] After identifying the two levels of key pixel clusters, the system overlays rendering onto the original solder joint image. For example, the system highlights all areas belonging to the first key pixel cluster with a semi-transparent bright red. This striking color immediately attracts the attention of reviewers, clearly indicating that these are the most severely affected areas. Simultaneously, the system highlights all areas belonging to the second key pixel cluster with a different color, such as a semi-transparent yellow or orange. This color distinction allows reviewers to easily distinguish between the core and secondary related areas of the fault. The final output image with dual-color markings, along with diagnostic conclusions such as poor solder joints and a reliability score of 0.85, constitutes a complete and well-supported diagnostic report, significantly improving communication efficiency and decision-making accuracy.
[0076] Secondly, see Figure 2 This application also provides an image recognition-based electronic component solder joint fault diagnosis system. This system is the physical carrier and execution entity of the aforementioned method, and its internal structure and functional modules correspond one-to-one with the method steps. The system includes: Image acquisition and preprocessing module 210 is used to acquire solder joint images of the electronic components to be diagnosed and to preprocess the solder joint images. The preliminary judgment module 220 is used to locate the solder joint area in the preprocessed solder joint image, extract the image features of the solder joint area, make a preliminary judgment on the solder joint fault type based on the extracted image features, and calculate the reliability value of the preliminary judgment result. The disturbance analysis module 230 is used to divide the solder joint area into multiple pixel sub-regions and perform a disturbance operation on each pixel sub-region in turn. The disturbance operation is used to simulate the changes in image information within the pixel sub-region. The secondary judgment module 240 is used to make a secondary judgment on the solder joint fault type based on the image features of the solder joint image after each disturbance operation, and calculate the reliability value of the secondary judgment result. The difference between the reliability value of the preliminary judgment result and the reliability value of the secondary judgment result is used as the contribution value of the pixel sub-region. The key pixel cluster identification module 250 is used to identify at least one key pixel cluster from all pixel sub-regions based on the magnitude of the contribution value, and to visually mark the key pixel cluster on the solder joint image as visual evidence for fault diagnosis.
[0077] This technical solution provides a physical system for implementing the aforementioned diagnostic method. Through modular functional division, the complex diagnostic process is broken down into clear and independent execution units. The structure is clear, easy to implement and maintain, and provides reliable hardware and software architecture support for the practical application of this diagnostic method.
[0078] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for diagnosing solder joint faults in electronic components based on image recognition, characterized in that, The method includes: Acquire solder joint images of the electronic components to be diagnosed, and preprocess the solder joint images; In the preprocessed solder joint image, the solder joint area is located and the image features of the solder joint area are extracted. Based on the extracted image features, the solder joint fault type is preliminarily determined and the reliability value of the preliminary determination result is calculated. The solder joint area is divided into multiple pixel sub-regions, and a perturbation operation is performed on each pixel sub-region in sequence. The perturbation operation is used to simulate the change of image information within the pixel sub-region. After each disturbance operation, the solder joint fault type is determined a second time based on the image features of the solder joint image after the disturbance operation, and the reliability value of the second determination result is calculated. The difference between the reliability value of the initial determination result and the reliability value of the second determination result is used as the contribution value of the pixel sub-region. Based on the magnitude of the contribution value, at least one key pixel cluster is identified from all the pixel sub-regions, and the key pixel cluster is visually marked on the solder joint image as visual evidence for fault diagnosis.
2. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The step of preprocessing the solder joint image includes: Calculate the global average brightness value of the solder joint image, and perform brightness correction on the solder joint image based on the comparison result between the global average brightness value and the preset target brightness value; The solder joint image is noise-removed by median filtering, and the brightness value of each pixel in the solder joint image is replaced with the median of the brightness values of a preset pixel area in its neighborhood.
3. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The step of locating the solder joint region in the preprocessed solder joint image includes: Obtain the preset standard solder joint grayscale template; In the preprocessed solder joint image, the normalized cross-correlation coefficient between each window region and the preset standard solder joint grayscale template is calculated using a preset sliding window. The region with the normalized cross-correlation coefficient higher than a preset threshold is identified as a candidate region, and the region with the highest normalized cross-correlation coefficient among the candidate regions is selected as the solder joint region.
4. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The image features include geometric shape features characterizing the geometry of the solder joint, texture features characterizing the microstructure of the solder joint surface, and brightness distribution features characterizing the brightness distribution of the solder joint.
5. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 4, characterized in that, The step of making a preliminary determination of the solder joint fault type based on the extracted image features and calculating the reliability value of the preliminary determination result includes: According to the preset fault determination rules, the geometric shape features, the texture features, and the brightness distribution features are compared with the corresponding preset determination thresholds to determine the solder joint fault type; The degree of deviation of the geometric shape feature, the texture feature, and the brightness distribution feature from the corresponding preset judgment threshold is calculated, and the reliability value is generated by weighted calculation based on each degree of deviation.
6. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The step of sequentially performing a perturbation operation on each of the pixel sub-regions includes: The brightness values of all pixels within the pixel sub-region are sequentially replaced with the average brightness value of the neighboring pixels surrounding the pixel sub-region.
7. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The step of sequentially performing a perturbation operation on each of the pixel sub-regions includes: Call the preset background reference library containing a clean and flawless background texture; The surrounding environment texture of the pixel sub-region is extracted sequentially, and a matching reference texture is retrieved from the background reference library; The image information of the pixel sub-region is replaced with the retrieved reference texture.
8. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 1, characterized in that, The step of identifying at least one key pixel cluster from all the pixel sub-regions based on the magnitude of the contribution value includes: Sort all the pixel sub-regions in descending order of their contribution values; The pixel sub-regions whose contribution values are ranked in the first preset first number of positions are determined as the first key pixel cluster; The pixel sub-regions whose contribution values are sorted after the first preset number of positions and then the second preset number of positions are determined as the second key pixel cluster.
9. The method for diagnosing solder joint faults in electronic components based on image recognition according to claim 8, characterized in that, The step of visually marking the key pixel clusters on the solder joint image as visual evidence for fault diagnosis includes: The first key pixel cluster is highlighted with a first color, and the second key pixel cluster is highlighted with a second color.
10. An image recognition-based electronic component solder joint fault diagnosis system, used to execute the image recognition-based electronic component solder joint fault diagnosis method as described in any one of claims 1 to 9, characterized in that, The system includes: The image acquisition and preprocessing module is used to acquire solder joint images of the electronic components to be diagnosed and to preprocess the solder joint images. The preliminary judgment module is used to locate the solder joint area in the preprocessed solder joint image, extract the image features of the solder joint area, make a preliminary judgment on the solder joint fault type based on the extracted image features, and calculate the reliability value of the preliminary judgment result. The perturbation analysis module is used to divide the solder joint area into multiple pixel sub-regions, and perform a perturbation operation on each pixel sub-region in sequence. The perturbation operation is used to simulate the change of image information within the pixel sub-region. The secondary determination module is used to perform a secondary determination of the solder joint fault type based on the image features of the solder joint image after each disturbance operation, and to calculate the reliability value of the secondary determination result. The difference between the reliability value of the preliminary determination result and the reliability value of the secondary determination result is used as the contribution value of the pixel sub-region. The key pixel cluster identification module is used to identify at least one key pixel cluster from all the pixel sub-regions according to the magnitude of the contribution value, and to visually mark the key pixel cluster on the solder joint image as visual evidence for fault diagnosis.