Welding spot degradation state detection method, device and equipment and storage medium

Abstract

The embodiment of the invention relates to the field of defect detection, and discloses a welding spot defect detection method, device and equipment and a storage medium, and the method comprises the steps: obtaining at least two welding spot images of the same welding spot on a to-be-detected circuit board, and enabling the welding spot images to have different vertical heights from the welding spot during shooting; fusing the welding spot images to obtain a fused image; inputting the fused image into a defect identification model for defect identification, and outputting the category of the defect and the area where the defect is located; determining form information of the defect based on the area where the defect is located; and determining the degradation state of the welding spot according to the form information of the defect and the type of the defect. According to the application, the degradation state of the welding spot is detected, and the accuracy of evaluating the degradation state of the welding spot is improved, so that the work such as life evaluation and maintenance of a circuit board is effectively carried out.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of defect detection, and particularly to a method, apparatus, equipment, and storage medium for detecting solder joint defects. Background Technology

[0002] With the rapid development of electronic packaging technology, components are becoming increasingly miniaturized and integrated, with smaller and more numerous internal solder joints. The main function of these internal solder joints is mechanical support and electrical connection; if even one solder joint breaks, it can cause the entire component to malfunction or fail, leading to serious consequences. Effectively assessing the degradation status of each solder joint during electronic circuit board maintenance can reduce the failure rate of products in the field and avoid the waste of resources caused by premature product replacement, thereby improving the reliability of electronic circuit boards.

[0003] Traditional methods for assessing weld joint degradation, such as manual visual inspection with cross-sectioning, and electrical and strain parameter testing, cannot meet the demands of non-destructive, high-efficiency, high-accuracy, and standardized maintenance scenarios. In recent years, with the development of image recognition and deep learning technologies, these technologies have been increasingly applied to weld joint defect detection devices. However, these devices primarily identify defects from the production stage, and the detection of fine cracks caused by stress during the service life phase has not been thoroughly studied. Summary of the Invention

[0004] The purpose of this invention is to provide at least one method, apparatus, device, and storage medium for detecting the degradation state of solder joints, which can at least solve the technical problem of the inability to accurately detect solder joint defects.

[0005] To address the aforementioned technical problems, at least one embodiment of this application provides a method for detecting the degradation state of solder joints, comprising:

[0006] Acquire at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image is different when it is taken;

[0007] The images of each solder joint are fused to obtain a fused image;

[0008] The fused image is input into the defect recognition model for defect recognition, and the defect category and the region where the defect is located are output.

[0009] The morphological information of the defect is determined based on the region where the defect is located;

[0010] The degradation state of the solder joint is determined based on the morphological information and the category of the defect.

[0011] At least one embodiment of this application also provides a solder joint degradation state detection device, comprising:

[0012] The acquisition module is used to acquire at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image from the solder joint is different when it is captured;

[0013] The fusion module is used to fuse the images of each solder joint to obtain a fused image;

[0014] The defect identification module is used to input the fused image into the defect identification model for defect identification, and output the category of the defect and the area where the defect is located;

[0015] An extraction module is used to determine the morphological information of the defect based on the region where the defect is located;

[0016] The detection module is used to determine the degradation state of the solder joint based on the morphological information and the category of the defect.

[0017] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described solder joint degradation state detection method.

[0018] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described solder joint degradation state detection method.

[0019] The solder joint degradation state detection method provided in the embodiments of this application, on the one hand, fuses solder joint images obtained from different heights to obtain a fused image of the solder joint, ensuring that the clarity of the obtained fused image is high enough to facilitate subsequent identification of solder joint defects, thus solving the technical problem that the solder joint images obtained in the prior art cannot accurately reflect the morphological information of solder joint defects because the solder joints on the circuit board are too small; on the other hand, by inputting the fused image into a defect identification model for defect identification, the defect category and the defect location area are determined, defect morphological information is extracted based on the defect location area, and the degradation state of the solder joint is determined based on the defect category and defect morphological information, thereby improving the accuracy of solder joint degradation state assessment, so as to effectively carry out circuit board life assessment, maintenance and other work.

[0020] In some optional embodiments, the step of fusing the images of each solder joint to obtain a fused image includes:

[0021] Each of the solder joint images is divided equally to obtain multiple original image blocks for each solder joint image;

[0022] For each of the solder joint images, a discrete cosine transform is performed on each of the original image blocks to obtain the transformed image blocks.

[0023] Calculate the focus of each of the transformed image blocks, and determine the transformed image block with the largest focus in the same region among the solder joint images as the candidate focus image block for that region;

[0024] The selected focus image blocks from each region are stitched together to obtain a fused image of the weld joint.

[0025] In this way, the focus of each solder joint image is calculated by dividing the image into blocks. The image block with the highest focus in the same region of each solder joint image is selected as the candidate focus image block. That is, the image block that reflects the richest edge and detail information in the same region of each solder joint image is filtered out and determined as the candidate focus image block. By stitching together the candidate focus image blocks of each region, a fused image of the solder joint can be obtained. The fused image of the solder joint obtained at this time can have a large depth of field while maintaining high definition, which facilitates the subsequent identification of defects in the solder joint.

[0026] In some optional embodiments, the solder joint degradation state includes at least two of normal, degraded, and failed, and each solder joint degradation state corresponds to a preset threshold range; the step of determining the solder joint degradation state based on the defect morphology information and the defect category includes:

[0027] The solder joint degradation score is calculated based on the defect morphology information and the defect category.

[0028] The solder joint degradation state is determined based on the comparison result between the solder joint degradation score and the preset threshold range corresponding to the solder joint degradation state.

[0029] The determination of the degradation state of solder joints based on the category identification results and morphological information of solder joint defects improves the accuracy of solder joint degradation state assessment.

[0030] In some optional embodiments, the step of determining the morphological information of the defect based on the region where the defect is located includes:

[0031] Skeleton extraction is performed on the area where the defect is located to obtain the morphological information of the defect.

[0032] By converting crack images into skeleton images, the geometric features of cracks, such as length, direction, and branching, can be better described and analyzed. This information is crucial for understanding the characteristics of cracks and conducting subsequent structural safety assessments.

[0033] In some optional embodiments, the step of acquiring at least two solder joint images of the circuit board under test further includes:

[0034] Determine the location of solder joints on the circuit board under test. This ensures accurate identification of solder joint defects later.

[0035] In some optional embodiments, the step of determining the location information of solder joints on the circuit board under test includes:

[0036] Obtain the original image of the circuit board under test;

[0037] Determine the calibration points of the original image and the locations of the solder joints in the original image;

[0038] The location of the solder joint on the circuit board under test in physical space is determined based on the relative position of the calibration point of the original image and the location of the solder joint in the original image.

[0039] In this way, high-definition images of solder joint defects were acquired in a cost-effective manner, saving image storage costs while improving the accuracy of the defect recognition algorithm.

[0040] In some optional embodiments, the step of inputting the fused image into a defect recognition model for defect recognition and outputting the defect category and the region where the defect is located further includes:

[0041] Solder joint defects are classified into at least two categories;

[0042] Acquire sample solder joint images, including solder joint defects and corresponding annotations for the categories of solder joint defects;

[0043] An initial defect identification model is constructed based on an instance segmentation algorithm. The sample solder joint image is then input into the initial defect identification model for optimization training to obtain the defect identification model.

[0044] A defect identification model is constructed based on an instance segmentation algorithm. This model identifies whether cracks exist in weld point images and outputs the coordinates of the crack area at the pixel level, which facilitates the subsequent extraction of defect morphology information from the crack area coordinates. Attached Figure Description

[0045] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0046] Figure 1 This is a flowchart of a solder joint degradation state detection method provided in one embodiment of this application;

[0047] Figure 2This is a flowchart of a solder joint degradation state detection method provided in another embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a solder joint degradation state detection device provided in another embodiment of this application;

[0049] Figure 4 This is a structural block diagram of a solder joint degradation state detection device provided in another embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0052] This invention proposes a method for detecting the degradation state of circuit board solder joints. The implementation details of the shale pore structure detection method in this embodiment are described below. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0053] Example 1:

[0054] The solder joint degradation detection method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes:

[0055] Step 110: Obtain at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image is different when it is taken.

[0056] Specifically, after determining the location of the solder joints on the circuit board under test, an optical system is used to acquire images of the solder joints. By changing the relative vertical height between the optical system and the solder joint, images of the same solder joint at different height positions can be obtained.

[0057] In some examples, at least two images of the same solder joint on the circuit board under test are obtained from a database, wherein the vertical height of each solder joint image is different when it is taken.

[0058] Step 120: Fuse the images of each solder joint to obtain a fused image.

[0059] Specifically, because the solder joints on circuit boards are typically only a few millimeters in size, and solder joint defects, such as cracks, are on the order of micrometers in width, and because the solder joint surface has a certain height, the solder joint images directly obtained in existing technologies are affected by their physical characteristics, making it difficult to maintain high definition while simultaneously achieving a sufficiently large depth of field. Depth of field refers to the range of distances in front of and behind a photographed object that can form a sharp image at the front of a lens or other imaging device. In simpler terms, objects at the focal plane will form a sharp image on the film, and there is a range from a certain point in front of the focal plane to a certain point behind it where objects can form a sharp image; this range is called the depth of field.

[0060] In this embodiment, multiple images of the solder joint at different vertical heights are acquired, and the images are fused to ensure that the fused image maintains high clarity while having sufficient depth of field, which facilitates the subsequent identification of defects in the solder joint.

[0061] Step 130: Input the fused image into the defect recognition model for defect recognition, and output the defect category and the region where the defect is located.

[0062] In this embodiment, a crack in the solder joint is used as an example for illustration. Specifically, based on engineering experience, the cracks in the solder joint are divided into at least two categories. The solder joint images are labeled with different categories. A defect recognition model is trained using a dataset that includes the solder joint images and the corresponding labels for the cracks in the solder joints. The defect recognition model is then used to identify defects in the fused image and output the category of the defect and the region where the defect is located.

[0063] In some cases, the cracks in the solder joints are classified into three categories based on the size of the cracks: micro-cracks, small cracks, and large cracks. The instance segmentation model is trained using a dataset of multiple sets of solder joint images that include micro-cracks, small cracks, and large cracks, thereby obtaining a defect recognition model.

[0064] In some cases, the instance segmentation model chosen is the YOLO (You Only Look Once) model. The YOLO model is an object detection and image segmentation model that can not only detect objects, but also perform instance segmentation tasks, that is, identify and segment individual crack instances in an image to obtain the pixel-level coordinates of the crack region. These coordinates are used to indicate which pixels in the image belong to the crack.

[0065] Step 140: Determine the morphological information of the defect based on the region where the defect is located.

[0066] Specifically, the region where the defect is located is segmented based on the instance segmentation results to obtain a segmented individual defect image, and the defect morphology information is extracted from the individual defect image.

[0067] In some examples, the weld defect is described as a crack, and the morphological information of the defect includes the length and width of the crack.

[0068] Step 150: Determine the degradation state of the solder joint based on the morphological information and the category of the defect.

[0069] Specifically, the degradation score of the solder joint is determined by the morphological information and category of the defect, and the degradation state of the solder joint is determined based on the degradation score. Each degradation state of the solder joint corresponds to a preset threshold range.

[0070] In some cases, the degradation score of a solder joint is determined by weighting the morphological information and the type of defect.

[0071] In some cases, by establishing a neural network prediction model, the morphological information and category of the defect are input into the neural network prediction model to predict the degradation score of the solder joint.

[0072] In some cases, the degradation state of a solder joint is characterized by different values ​​or different levels to indicate the severity of the degradation.

[0073] In this embodiment, on the one hand, the solder joint images obtained from different heights are fused to obtain a fused image of the solder joint. This ensures that the clarity of the fused image is sufficiently high, facilitating subsequent identification of solder joint defects. This addresses the technical problem that solder joint images obtained in the prior art cannot accurately reflect the morphological information of solder joint defects because the solder joints on the circuit board are too small. On the other hand, by inputting the fused image into a defect identification model for defect identification, the defect category and the defect location are determined. Defect morphological information is extracted based on the defect location. Based on the defect category and defect morphological information, the degradation state of the solder joint is determined, improving the accuracy of the solder joint degradation state assessment. This enables effective life assessment and repair of the circuit board.

[0074] In some embodiments, the step of fusing the images of each solder joint to obtain a fused image includes:

[0075] Each of the solder joint images is divided equally to obtain multiple original image blocks for each solder joint image;

[0076] For each of the solder joint images, a discrete cosine transform is performed on each of the original image blocks to obtain the transformed image blocks.

[0077] Calculate the focus of each of the transformed image blocks, and determine the transformed image block with the largest focus in the same region among the solder joint images as the candidate focus image block for that region;

[0078] The selected focus image blocks from each region are stitched together to obtain a fused image of the weld joint.

[0079] Specifically, the solder joint image is divided into segments, for example, using 32x32 squares. A Discrete Cosine Transform (DCT) is then performed on each square of the solder joint image to obtain transformed image blocks. The DCT is a transformation related to the Fourier Transform, specifically converting a time-domain signal (such as an image) into a frequency-domain representation. Through the DCT, a signal is transformed from the time domain to the frequency domain, revealing its energy distribution at different frequencies. In this embodiment, the squares after the DCT are transformed into the DCT domain, facilitating subsequent image processing.

[0080] Focus is a key indicator of whether objects or scenes in an image are clearly presented. It mainly depends on the visibility of details in the image, especially the sharpness of edges and textures.

[0081] In some examples, the Energy of Laplace (EOL) is used to calculate the focus of the transformed image patch. Image sharpness is evaluated by calculating the second derivative of the image (i.e., the Laplacian operator). A higher EOL value generally indicates better image focus. The Laplacian energy is an image energy value calculated based on the Laplacian operator. The Laplacian operator highlights rapidly changing areas in the image, such as edges and details. The magnitude of the EOL value reflects the richness of edge and detail information in the image.

[0082] The focus of each transformed image block in each solder joint image is calculated. The transformed image block with the highest focus in the same region in each solder joint image is determined as the candidate focused image block for that region. In other words, the transformed image block that reflects the richest edge and detail information in the image is selected and determined as the candidate focused image block.

[0083] By stitching together the candidate focused image blocks from each region, a fused image of the solder joint can be obtained. This fused image maintains high resolution while possessing a large depth of field, facilitating subsequent defect identification of the solder joint. In this embodiment, high-definition solder joint defect images are acquired in a cost-effective manner, saving image storage costs while improving the accuracy of defect identification.

[0084] In some embodiments, the solder joint degradation state includes at least two of normal, degraded, and failed, and each solder joint degradation state corresponds to a preset threshold range; the step of determining the solder joint degradation state based on the defect morphology information and the defect category includes:

[0085] The solder joint degradation score is calculated based on the defect morphology information and the defect category.

[0086] The solder joint degradation state is determined based on the comparison result between the solder joint degradation score and the preset threshold range corresponding to the solder joint degradation state.

[0087] In one example, the fused image is input into a defect recognition model for defect prediction. The defect recognition model outputs the defect category, the region where the defect is located, and the confidence level. The morphological information of the defect is determined based on the defect region, and the weld degradation score is determined by weighted calculation based on the defect morphological information, crack category, and confidence level.

[0088] Specifically, it includes:

[0089] Step 11: Assign different weights to the crack length, crack width, model category output by the model, and corresponding confidence level according to different crack categories, and perform comprehensive calculation to obtain the final degradation value.

[0090] Step 12: Pre-set thresholds according to different degradation states of the weld joint. The degradation state is divided into three categories: normal, degradation, and failure. When the degradation value is less than the first preset threshold, the degradation state corresponding to the crack category is determined to be normal. When the degradation value is greater than the first preset threshold but less than the second preset threshold, the degradation state corresponding to the crack category is determined to be degradation. When the degradation value is greater than the second preset threshold, the degradation state corresponding to the crack category is determined to be failure. The first preset threshold is less than the second preset threshold, and the second threshold is less than the third preset threshold.

[0091] In one example, the weld degradation score is determined using a neural network based on morphological information and crack detection information, including:

[0092] Step 21: Data collection and solder joint degradation scoring.

[0093] Specifically, samples of various solder joints are obtained, with each sample's solder joint status covering all states from fully functional, degraded, and failed. Each solder joint is scored with a degradation score, which can be a continuous numerical range (e.g., 0-100), where 0 represents fully functional and 100 represents completely failed.

[0094] Step 22: Degradation threshold setting.

[0095] Based on the scoring results, a threshold for the degradation score is determined. In one example, the degradation score threshold includes a first preset threshold and a second preset threshold. When the degradation score is less than the first preset threshold, the degradation state of the solder joint is determined to be normal. When the degradation score is greater than the first preset threshold but less than the second preset threshold, the degradation state of the solder joint is determined to be degraded. When the degradation score is greater than the second preset threshold, the degradation state of the solder joint is determined to be failed. For example, the degradation state of solder joints with a degradation score below 30 can be classified as normal, the degradation state of solder joints with a degradation score between 30 and 70 can be classified as degraded, and the degradation state of solder joints with a degradation score greater than 70 can be classified as failed.

[0096] Step 23: Training the neural network model.

[0097] Specifically, a suitable neural network architecture, such as a multilayer perceptron (MLP) or a convolutional neural network (CNN), is selected based on the number and complexity of the input features. The output layer of the neural network is set to a single neuron, and the output value is the degradation score (corresponding to the expert score). Cracks in the weld joints are classified and assigned integer values ​​(such as crack type 1, crack type 2, etc.). The crack classification value corresponding to the crack of each weld joint, as well as the width and length of the crack, are used as input features of the neural network model and input into the model for training.

[0098] Step 24: Application of the model:

[0099] The solder joints whose degradation status needs to be determined are input into the trained neural network model, and the predicted degradation score is output.

[0100] Based on the degradation threshold preset in step 22, the predicted degradation score is compared with the threshold to determine the degradation state of the solder joint.

[0101] For example, the crack classification value corresponding to a crack in a weld joint, as well as the width and length of the crack, are input into a neural network model. The neural network model outputs a predicted degradation score of 50. If the predicted degradation score is greater than a first preset threshold and less than a second preset threshold, then the degradation state of the weld joint is determined to be degradation. This achieves the assessment of the degradation state of the weld joint.

[0102] In this embodiment, the degradation state of the solder joint is judged based on the category identification result of the solder joint defect and the morphological information of the defect, which improves the accuracy of the assessment of the degradation state of the solder joint.

[0103] In some embodiments, the step of determining the morphological information of the defect based on the region where the defect is located includes:

[0104] Skeleton extraction is performed on the area where the defect is located to obtain the morphological information of the defect.

[0105] Specifically, taking weld point defects as an example, we will illustrate how a crack skeleton extraction algorithm can be used to extract the morphological information of the crack from the weld point defect. The main goal of the crack skeleton extraction algorithm is to extract the skeleton shape of the crack from the image. By converting the crack image into a skeleton image, we can better describe and analyze the geometric features of the crack, such as its length, direction, and branching. This information is crucial for understanding the characteristics of the crack and conducting subsequent structural safety assessments.

[0106] In some examples, the steps for skeleton extraction of the defect area include: converting the image of the defect area to grayscale; using different filters and enhancement algorithms to remove noise and unnecessary details from the image to obtain a clearer crack image; during crack skeletonization, morphological operations can be applied to the crack image to extract the main structure of the crack, i.e., the crack skeleton. Morphological operations are based on erosion and dilation of structuring elements and the image. By iteratively performing erosion and dilation operations, the crack edges can be gradually contracted, ultimately extracting the crack skeleton structure. Post-processing of the extracted crack skeleton, such as removing burrs and filling small holes, can improve image quality and obtain more accurate crack morphology information.

[0107] In some cases, the weld joint image is too complex to be directly binarized to extract the crack. Therefore, it is necessary to segment the crack area according to the instance segmentation structure and eliminate the background influence. That is, in the first step, the defect is identified by the defect identification model, and the defect category and the area where the defect is located are output. Then, the crack morphology information is extracted by image binarization, image denoising and crack skeleton map.

[0108] In some cases, crack morphology information includes crack length, width, etc.

[0109] In some embodiments, the step of acquiring at least two solder joint images of the circuit board under test further includes:

[0110] Determine the location information of the solder joints on the circuit board under test.

[0111] Specifically, in order to ensure accurate identification of solder joint defects, it is also necessary to determine the location information of the solder joints on the circuit board under test.

[0112] In some cases, an image of the circuit board under test is first acquired, and the location of the solder joints on the image is determined. The solder joint locations are then converted into their actual physical locations to determine the location information of the solder joints.

[0113] In some embodiments, the step of acquiring at least two solder joint images of the same solder joint on the circuit board under test includes:

[0114] Obtain the original image of the circuit board under test;

[0115] Determine the calibration points of the original image and the locations of the solder joints in the original image;

[0116] The location of the solder joint on the circuit board under test in physical space is determined based on the relative position of the calibration point of the original image and the location of the solder joint in the original image.

[0117] Specifically, an optical system is used to acquire images of the circuit board to obtain the original image of the circuit board under test. Then, an image processing and detection system is used to identify the location of all solder joints. In this process, a template matching algorithm is used to find the calibration points of the circuit board in the image, and then the location of the solder joints in the original image is determined based on the relative position of the calibration points and the solder joints. Based on the location information of the solder joints in the image, the location information of the solder joints in physical space is inferred.

[0118] In some examples, to facilitate the acquisition of at least two images of the same solder joint on the circuit board under test, the method further includes: planning the shooting order of the solder joints based on their location information. Further, based on the location information of the solder joints, a genetic algorithm is used to determine the optimal shooting order and shooting path. According to the set shooting path, a robotic arm connected to the optical system is moved, thereby moving the optical system to positions above the solder joint at varying vertical heights for image acquisition.

[0119] In some embodiments, the step of inputting the fused image into a defect recognition model for defect recognition and outputting the defect category and the region where the defect is located further includes:

[0120] Solder joint defects are classified into at least two categories;

[0121] Acquire sample solder joint images, including solder joint defects and corresponding annotations for the categories of solder joint defects;

[0122] An initial defect identification model is constructed based on an instance segmentation algorithm. The sample solder joint image is then input into the initial defect identification model for optimization training to obtain the defect identification model.

[0123] Specifically, based on engineering experience, the cracks in the weld joints are divided into at least two categories. The weld joint images are labeled with a dataset according to the different categories. The labeling information includes the location (boundary box coordinates) and category of the crack. The defect recognition model is trained using the dataset that includes the weld joint images and the corresponding labeling information of the cracks in the weld joints. The defect recognition model is used to identify defects in the fused images and outputs the category of the defect and the region where the defect is located.

[0124] In some examples, the instance segmentation model uses the YOLO (You Only Look Once) model. YOLO is an object detection and image segmentation model that can not only detect objects but also perform instance segmentation tasks, i.e., identify and segment individual crack instances in an image to obtain pixel-level coordinates of the crack region. These coordinates are used to indicate which pixels in the image belong to the crack. Furthermore, the instance segmentation model uses the YOLO v8 model, an improved YOLO model that not only supports object detection but also, through improvements, can perform instance segmentation tasks, i.e., identify and segment individual crack instances in an image, offering higher detection accuracy and speed. Specifically, by inputting the fused image of the weld joint into the YOLO v8 model, crack defects in the weld joint are detected, and the crack region coordinates are output at the pixel level.

[0125] In some cases, due to the complex background conditions of the weld joints, there are many false detections. Based on the analysis of crack geometry features, the false detection of cracks is optimized to remove falsely detected cracks such as scratches and stains.

[0126] In some examples, the fused image is input into a defect recognition model for defect prediction. The defect recognition model outputs the defect category, the defect location, and the confidence level. The morphological information of the defect is determined based on its location, and the degradation state of the weld is determined based on the morphological information, the defect category, and the confidence level. Specifically, the degradation state is assessed through a weighted calculation based on the defect morphological information, crack category, and confidence level.

[0127] Example 2:

[0128] The circuit board solder joint defect identification and degradation status assessment method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 2 As shown, it includes:

[0129] Step 201: Obtain the circuit board image and use the image processing detection system to identify the location of all solder joints.

[0130] Specifically, it includes the following steps:

[0131] 1a. First, place the electronic board (i.e., circuit board) into the stage and fix it with a clamp. The software will control the optical system ① to acquire a complete image of the electronic board.

[0132] 1b. Use template matching algorithm to find the calibration points of the electronic board in the image, and then determine the position of the solder joint in the image based on the relative position information of the calibration points and the solder joints;

[0133] 1c. Based on the location information of the solder joints in the image, calculate the location information of the solder joints in physical space and save the relevant results.

[0134] Step 202: Plan the shooting sequence of the solder joints, use a robotic arm to move the camera to acquire solder joint images, and fuse the multi-focus images.

[0135] Specifically, it includes the following steps:

[0136] 2a. Based on the location information of the solder joints, use a genetic algorithm to determine the optimal shooting order and shooting path;

[0137] 2b. According to the set shooting path, move the robotic arm to move the optical system ② directly above the solder joint. The robotic arm will move the optical system ② to four different heights to collect images of the solder joint.

[0138] 2c. Then, the image is divided into four images using 32x32 small squares. The small squares are transformed to the DCT domain using DCT transformation. The focus of the image is then evaluated using EOL (Laplacian Energy Function). The small square with the best focus evaluation among the four images in the same region is selected as part of the final fully focused image, thus obtaining the complete fully focused image, which is then input into the image processing and detection system.

[0139] The Discrete Cosine Transform (DCT) is a transformation related to the Fourier Transform. Specifically, it converts a time-domain signal (such as an image) into a frequency-domain representation. Through the DCT transformation, a signal is transformed from the time domain to the frequency domain, revealing its energy distribution at different frequencies. In this embodiment, the small squares after the Discrete Cosine Transform are transformed into the DCT domain for easier subsequent image processing.

[0140] Focus is a key indicator of whether objects or scenes in an image are clearly presented. It mainly depends on the visibility of details in the image, especially the sharpness of edges and textures.

[0141] In some examples, the Energy of Laplace (EOL) is used to calculate the focus of the transformed image patch. Image sharpness is evaluated by calculating the second derivative of the image (i.e., the Laplacian operator). A higher EOL value generally indicates better image focus. The Laplacian energy is an image energy value calculated based on the Laplacian operator. The Laplacian operator highlights rapidly changing areas in the image, such as edges and details. The magnitude of the EOL value reflects the richness of edge and detail information in the image.

[0142] Because the solder joints on circuit boards are typically only a few millimeters in size, and solder joint defects, such as cracks, are on the order of micrometers in width, and because the solder joint surface has a certain height, the solder joint images directly obtained in current technologies are affected by their physical characteristics, making it difficult to maintain high clarity while simultaneously achieving a sufficiently large depth of field. Depth of field refers to the range of distances in front of and behind a photographed object that can form a clear image at the front of a lens or other imaging device. Simply put, objects at the focal plane will form a clear image on the film, and there is a range from a certain point in front of the focal plane to a certain point behind it where objects can form a clear image; this range is called the depth of field.

[0143] In this embodiment, multiple images of the solder joint at different vertical heights are acquired, and the images are fused to ensure that the fused image maintains high clarity while having sufficient depth of field, which facilitates the subsequent identification of defects in the solder joint.

[0144] Furthermore, in this embodiment, high-definition solder joint defect images are acquired in a cost-effective manner, saving image storage costs while improving the accuracy of the defect recognition algorithm.

[0145] Step 203: Use the defect detection model to identify defects and use the degradation detection model to determine the degradation state.

[0146] Specifically, it includes the following steps:

[0147] 3a. Based on the crack width, the present invention classifies cracks into micro-cracks, small cracks, and large cracks. First, the YOLOv8 crack instance segmentation algorithm is used to detect whether the image contains cracks, and the crack region coordinates are output at the pixel level.

[0148] 3b. Due to the complex background conditions of the weld joints, there are many false detections. Based on the analysis of crack geometry features, crack false detection is optimized to remove cracks that are falsely detected due to scratches, stains, etc.

[0149] 3c. Since the weld joint image is complex and cannot be directly binarized to extract the crack, it is necessary to segment the crack area according to the instance segmentation result, eliminate the background influence, and then obtain the crack morphology information, mainly including crack length, width and other information, through image binarization, image denoising and crack skeleton map extraction.

[0150] 3d. Finally, based on morphological information (length, width, etc.) and crack detection information (crack classification, detection reliability), degradation state assessment is achieved through weighted calculation or neural network.

[0151] In some cases, based on engineering experience, the cracks in the solder joints are classified into at least two categories. The solder joint images are labeled with a dataset according to the different categories. The labeling information includes the location (boundary box coordinates) and category of the crack. The defect recognition model is trained using the dataset that includes the solder joint images and the corresponding labeling information of the cracks in the solder joints. The defect recognition model is used to identify defects in the fused images and outputs the category of the defect and the region where the defect is located.

[0152] As solder joints on a circuit board degrade, micro-cracks gradually appear on the surface. These micro-cracks expand over time, becoming small cracks, and then continue to expand into large cracks. In some cases, solder joint cracks are classified into three categories—micro-cracks, small cracks, and large cracks—based on their size. An instance segmentation model is trained using a dataset of solder joint images containing micro-cracks, small cracks, and large cracks, along with their corresponding annotations, to obtain a defect recognition model.

[0153] In some examples, the instance segmentation model uses the YOLO (You Only Look Once) model. YOLO is an object detection and image segmentation model that can not only detect objects but also perform instance segmentation tasks, i.e., identify and segment individual crack instances in an image to obtain pixel-level coordinates of the crack region. These coordinates are used to indicate which pixels in the image belong to the crack. Furthermore, the instance segmentation model uses the YOLO v8 model, an improved YOLO model that not only supports object detection but also, through improvements, can perform instance segmentation tasks, i.e., identify and segment individual crack instances in an image, offering higher detection accuracy and speed. Specifically, by inputting the fused image of the weld joint into the YOLO v8 model, crack defects in the weld joint are detected, and the crack region coordinates are output at the pixel level.

[0154] In some cases, due to the complex background conditions of the weld joints, there are many false detections. The crack area output by the model is preprocessed to remove falsely detected cracks such as scratches and stains. That is, crack false detection optimization is performed based on crack geometric feature analysis to delete falsely detected cracks such as scratches and stains.

[0155] In this process, the skeleton of the defect is extracted to obtain the morphological information of the defect.

[0156] Specifically, taking weld point defects as an example, we will illustrate how a crack skeleton extraction algorithm can be used to extract the morphological information of the crack from the weld point defect. The main goal of the crack skeleton extraction algorithm is to extract the skeleton shape of the crack from the image. By converting the crack image into a skeleton image, we can better describe and analyze the geometric features of the crack, such as its length, direction, and branching. This information is crucial for understanding the characteristics of the crack and conducting subsequent structural safety assessments.

[0157] In some examples, the steps for skeleton extraction of the defect area include: converting the image of the defect area to grayscale; using different filters and enhancement algorithms to remove noise and unnecessary details from the image to obtain a clearer crack image; during crack skeletonization, morphological operations can be applied to the crack image to extract the main structure of the crack, i.e., the crack skeleton. Morphological operations are based on erosion and dilation of structuring elements and the image. By iteratively performing erosion and dilation operations, the crack edges can be gradually contracted, ultimately extracting the crack skeleton structure. Post-processing of the extracted crack skeleton, such as removing burrs and filling small holes, can improve image quality and obtain more accurate crack morphology information.

[0158] In some cases, degradation status assessment can be achieved through weighted calculations or neural networks based on morphological information (length, width, etc.) and crack detection information (crack classification, detection reliability).

[0159] The degradation state is assessed through weighted calculation based on morphological information and crack detection information, including:

[0160] The fused image is input into the defect recognition model for defect prediction. The defect recognition model outputs the defect category, the defect location, and the confidence level. The morphological information of the defect is determined based on the defect location, and the degradation state is assessed through weighted calculation based on the defect morphological information, crack category, and confidence level.

[0161] Specifically, it includes:

[0162] Step 11: Assign different weights to the crack length, crack width, model category output by the model, and corresponding confidence level according to different crack categories, and perform comprehensive calculation to obtain the final degradation value.

[0163] Step 12: Pre-set thresholds according to different degradation states of the weld joint. The degradation state is divided into three categories: normal, degradation, and failure. When the degradation value is less than the first preset threshold, the degradation state corresponding to the crack category is determined to be normal. When the degradation value is greater than the first preset threshold but less than the second preset threshold, the degradation state corresponding to the crack category is determined to be degradation. When the degradation value is greater than the second preset threshold, the degradation state corresponding to the crack category is determined to be failure. The first preset threshold is less than the second preset threshold, and the second threshold is less than the third preset threshold.

[0164] Among them, the degradation state is assessed through a neural network based on morphological information and crack detection information, which includes:

[0165] Step 21: Data collection and solder joint degradation scoring.

[0166] Specifically, samples of various solder joints are obtained, with each sample's solder joint status covering all states from fully functional, degraded, and failed. Each solder joint is scored with a degradation score, which can be a continuous numerical range (e.g., 0-100), where 0 represents fully functional and 100 represents completely failed.

[0167] Step 22: Degradation threshold setting.

[0168] Based on the scoring results, a threshold for the degradation score is determined. In one example, the degradation score threshold includes a first preset threshold and a second preset threshold. When the degradation score is less than the first preset threshold, the degradation state of the solder joint is determined to be normal. When the degradation score is greater than the first preset threshold but less than the second preset threshold, the degradation state of the solder joint is determined to be degraded. When the degradation score is greater than the second preset threshold, the degradation state of the solder joint is determined to be failed. For example, the degradation state of solder joints with a degradation score below 30 can be classified as normal, the degradation state of solder joints with a degradation score between 30 and 70 can be classified as degraded, and the degradation state of solder joints with a degradation score greater than 70 can be classified as failed.

[0169] Step 23: Training the neural network model.

[0170] Specifically, a suitable neural network architecture, such as a multilayer perceptron (MLP) or a convolutional neural network (CNN), is selected based on the number and complexity of the input features. The output layer of the neural network is set to a single neuron, and the output value is the degradation score (corresponding to the expert score). Cracks in the weld joints are classified and assigned integer values ​​(such as crack type 1, crack type 2, etc.). The crack classification value corresponding to the crack of each weld joint, as well as the width and length of the crack, are used as input features of the neural network model and input into the model for training.

[0171] Step 24: Application of the model:

[0172] The solder joints whose degradation status needs to be determined are input into the trained neural network model, and the predicted degradation score is output.

[0173] Based on the degradation threshold preset in step 22, the predicted degradation score is compared with the threshold to determine the degradation state of the solder joint.

[0174] For example, the crack classification value corresponding to a crack in a weld joint, as well as the width and length of the crack, are input into a neural network model. The neural network model outputs a predicted degradation score of 50. If the predicted degradation score is greater than a first preset threshold and less than a second preset threshold, then the degradation state of the weld joint is determined to be degradation. This achieves the assessment of the degradation state of the weld joint.

[0175] In this embodiment, solder joint defect detection and degradation status identification based on solder joint appearance images are realized, which improves the accuracy of solder joint degradation status assessment. Compared with the method of assessment based on only a single piece of information, the accuracy is improved by more than 10%.

[0176] Example 3:

[0177] Another embodiment of this application relates to a solder joint degradation state detection device. The implementation details of this embodiment's solder joint degradation state detection device are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. A schematic diagram of the solder joint degradation state detection device in this embodiment can be seen as follows: Figure 3 As shown, it includes an acquisition module 801, a fusion module 802, a defect identification module 803, an extraction module 804, and a detection module 805.

[0178] The acquisition module 801 is used to acquire at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image from the solder joint is different when it is captured.

[0179] The fusion module 802 is used to fuse the images of each solder joint to obtain a fused image.

[0180] The defect recognition module 803 is used to input the fused image into the defect recognition model for defect recognition, and output the category of the defect and the area where the defect is located.

[0181] Extraction module 804 is used to determine the morphological information of the defect based on the region where the defect is located.

[0182] The detection module 805 is used to determine the degradation state of the solder joint based on the morphological information and the category of the defect.

[0183] Specifically, the acquisition module 801 includes optical system 1 and optical system 2, such as...Figure 4 As shown, optical system 1 acquires images of the entire electronic board (circuit board), mainly for locating solder joints; because the solder joints of the electronic board are tiny, optical system ② is also needed to acquire images of a single solder joint, which are used for defect identification.

[0184] The specific workflow of the acquisition module includes: placing the circuit board on a stage with fixtures to secure it; acquiring a complete image of the circuit board on the stage using optical system 1; processing the acquired image using an image processing system to automatically identify solder joint locations; and then moving optical system 2 directly above each solder joint using a robotic arm. At the same solder joint location, the robotic arm moves to multiple different heights to acquire images of the solder joint. These multiple partially focused images are then fused into a single fully focused image using a multi-focus image fusion algorithm. Finally, a defect detection algorithm is used to identify defects in the solder joints and determine their degradation status.

[0185] In one embodiment, the optical system acquires five images of the solder joints.

[0186] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0187] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0188] Example 4:

[0189] Another embodiment of this application relates to an electronic device, such as... Figure 5 As shown, it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the above method steps.

[0190] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0191] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0192] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component, for executing the methods in the above embodiments.

[0193] Example 5:

[0194] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the above-described method steps.

[0195] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for detecting the degradation state of solder joints, characterized in that, include: Acquire at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image is different when it is taken; The images of each solder joint are fused to obtain a fused image; The fused image is input into the defect recognition model for defect recognition, and the defect category and the region where the defect is located are output. The morphological information of the defect is determined based on the region where the defect is located; The degradation state of the solder joint is determined based on the morphological information and the category of the defect.

2. The method for detecting the degradation state of solder joints according to claim 1, characterized in that, The step of fusing the images of each solder joint to obtain a fused image includes: Each solder joint image is divided equally to obtain multiple original image blocks for each solder joint image; For each of the solder joint images, a discrete cosine transform is performed on each of the original image blocks to obtain the transformed image blocks. Calculate the focus of each of the transformed image blocks, and determine the transformed image block with the largest focus in the same region among the solder joint images as the candidate focus image block for that region; The selected focus image blocks from each region are stitched together to obtain a fused image of the weld joint.

3. The method for detecting the degradation state of solder joints according to claim 1, characterized in that, in, The solder joint degradation state includes at least two of the following: normal, degradation, and failure, and each solder joint degradation state corresponds to a preset threshold range; The step of determining the solder joint degradation state based on the defect morphology information and the defect category includes: The solder joint degradation score is calculated based on the defect morphology information and the defect category. The solder joint degradation state is determined based on the comparison result between the solder joint degradation score and the preset threshold range corresponding to the solder joint degradation state.

4. The method for detecting the degradation state of solder joints according to claim 1, characterized in that, The step of determining the morphological information of the defect based on the region where the defect is located includes: Skeleton extraction is performed on the area where the defect is located to obtain the morphological information of the defect.

5. The method for detecting the degradation state of solder joints according to claim 1, characterized in that, The step of acquiring at least two solder joint images of the circuit board under test also includes: Determine the location information of the solder joints on the circuit board under test.

6. The method for detecting the degradation state of solder joints according to claim 5, characterized in that, The step of determining the location information of solder joints on the circuit board under test includes: Acquire the original image of the circuit board under test; Determine the calibration points of the original image and the locations of the solder joints in the original image; The location of the solder joint on the circuit board under test in physical space is determined based on the relative position of the calibration point of the original image and the location of the solder joint in the original image.

7. The method for detecting the degradation state of solder joints according to claim 1, characterized in that, Before the step of inputting the fused image into the defect recognition model for defect recognition and outputting the defect category and the region where the defect is located, the following steps are also included: Solder joint defects are classified into at least two categories; Acquire sample solder joint images, including solder joint defects and corresponding annotations for the categories of solder joint defects; An initial defect identification model is constructed based on an instance segmentation algorithm. The sample solder joint image is then input into the initial defect identification model for optimization training to obtain the defect identification model.

8. A weld joint defect detection device, characterized in that, include: The acquisition module is used to acquire at least two images of the same solder joint on the circuit board under test, wherein the vertical height of each solder joint image from the solder joint is different when it is captured; The fusion module is used to fuse the images of each solder joint to obtain a fused image; The defect identification module is used to input the fused image into the defect identification model for defect identification, and output the category of the defect and the area where the defect is located; An extraction module is used to determine the morphological information of the defect based on the region where the defect is located; The detection module is used to determine the degradation state of the solder joint based on the morphological information and the category of the defect.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the solder joint degradation state detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the solder joint degradation state detection method according to any one of claims 1 to 7.

Images