Welding spot degradation state analysis method and system based on deep learning and storage medium

Abstract

The invention provides a welding spot degradation state analysis method and system based on deep learning and a storage medium, and the method comprises the following steps: collecting equipment information and welding spot external images of various sample equipment, and constructing a welding spot external image set in combination with the equipment information and the welding spot external images; according to the welding spot position information, welding spot positions are processed through a welding spot slicing method, and welding spot internal images of all sample devices are collected; constructing a welding spot degradation state set according to the internal images of the welding spots; constructing a welding spot degradation classification model based on the neural network model; training a welding spot degradation classification model by using the welding spot external image set and the welding spot degradation state set; inputting the target welding spot image into the trained welding spot degradation classification model to obtain a welding spot state classification result output by the welding spot degradation classification model; and determining the welding spot degradation state of the to-be-detected welding spot according to the welding spot state classification result. The device has the advantages that the detection efficiency is high and the device cannot be damaged when the degradation of the welding spot of the device is detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of equipment solder joint analysis, and specifically relates to a solder joint degradation state analysis method, system and storage medium based on deep learning. Background Art

[0002] With the rapid development of electronic packaging technology, components are becoming more and more miniaturized and integrated, and the size of internal solder joints is smaller and the number is larger. The main function of internal solder joints is mechanical support and electrical connection. Once a solder joint breaks, the entire component will malfunction or fail, resulting in serious consequences. If the degradation state of the solder joints of each electronic board can be evaluated during the maintenance process to determine whether the board needs to be replaced in advance, the risk of equipment failure can be greatly reduced and the reliability of the electronic board can be improved.

[0003] Traditional methods for evaluating the degradation status of solder joints include manual visual inspection plus slicing, and electrical parameter and strain parameter testing. However, the electrical measurement method is not very applicable to existing maintenance boards, strain parameters are difficult to obtain, and the change characteristics are not obvious. The manual visual inspection method has high requirements for maintenance personnel, and it also does not meet the maintenance needs of large quantities. The detection efficiency is low, and the slicing test is a destructive test, and the electronic boards cannot be used after cutting. Summary of the invention

[0004] The present invention provides a solder joint degradation state analysis method, system and storage medium based on deep learning to solve the problem that solder joint degradation detection efficiency is low and the detection process may cause equipment damage.

[0005] In a first aspect, the present invention provides a method for analyzing solder joint degradation state based on deep learning, the method comprising the following steps:

[0006] Collecting device information and external images of solder joints of multiple sample devices, and building a set of external images of solder joints by combining the device information and the external images of solder joints, wherein the device information includes solder joint position information;

[0007] Processing the solder joint positions of all the sample devices according to the solder joint position information and using a solder joint slicing method to collect internal images of the solder joints of all the sample devices;

[0008] constructing a solder joint degradation state set according to the internal image of the solder joint;

[0009] Constructing solder joint degradation classification model based on neural network model;

[0010] Training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set;

[0011] Acquire a target solder joint image of a solder joint to be tested in the device to be tested, input the target solder joint image into the trained solder joint degradation classification model, and obtain a solder joint state classification result output by the solder joint degradation classification model;

[0012] The solder joint degradation state of the solder joint to be tested is determined according to the solder joint state classification result.

[0013] Optionally, the step of processing the solder joint positions of all the sample devices by a solder joint slicing method according to the solder joint position information and collecting internal images of the solder joints of all the sample devices comprises the following steps:

[0014] According to the solder point position information, the solder point positions of the plurality of sample devices are cut into solder point slices according to preset sizes;

[0015] Inlaying the solder joint slices into a preset standard template to form a solder joint standard template;

[0016] Grinding and polishing the solder joint standard sample;

[0017] The image of the solder joint standard sample after grinding and polishing is collected by a solder joint image collection device as the internal image of the solder joint.

[0018] Optionally, the collecting of device information and external images of solder joints of multiple sample devices and combining the device information and the external images of solder joints to construct a set of external images of solder joints comprises the following steps:

[0019] Collect device information and external images of solder joints of various sample devices;

[0020] Extracting solder joint process information of each of the sample devices according to all the device information;

[0021] Retrieving a corresponding solder point feature set from a preset solder point feature database based on the solder point position information and the solder point process information;

[0022] Binding each solder joint feature in the solder joint feature set to the corresponding solder joint external image one by one;

[0023] A weld point external image set is constructed by combining all the weld point external images bound to the weld point features.

[0024] Optionally, constructing a weld point degradation state set according to the weld point internal image comprises the following steps:

[0025] Preprocessing the internal image of the solder joint to obtain a preprocessed solder joint image, wherein the preprocessing step includes an image enhancement step and an image rotation step;

[0026] Extracting solder joint degradation features from the preprocessed solder joint image using an image processing method;

[0027] Marking a degradation state for the internal image of the solder joint according to the solder joint degradation feature;

[0028] A solder joint degradation state set is constructed by combining all the solder joint internal images marked with the degradation state.

[0029] Optionally, the step of constructing a solder joint degradation classification model based on a neural network model includes the following steps:

[0030] Based on the neural network model, a solder joint feature detection model and a solder joint status classification model are constructed respectively;

[0031] Using the output layer of the solder joint feature detection model as the input layer of the solder joint state classification model;

[0032] The loss functions of the solder joint feature detection model and the solder joint state classification model are combined to obtain a comprehensive loss function, and the solder joint feature detection model and the solder joint state classification model are combined into a solder joint degradation classification model.

[0033] Optionally, the step of training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set comprises the following steps:

[0034] The solder joint degradation features and the solder joint degradation state set in the solder joint external image set are converted into a structured data set, and each of the solder joint degradation features in the structured data set has a corresponding feature value;

[0035] Establishing the relationship between the degradation characteristics of the solder joints and the degradation state in the structured data set by using a support vector machine;

[0036] The structured data set is used to perform quantitative analysis and training on the solder joint degradation classification model to optimize the loss function of the solder joint degradation classification model, where the loss function includes positioning loss, confidence loss and classification loss.

[0037] Optionally, the step of training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set comprises the following steps:

[0038] Performing standardized feature processing on the solder joint features in the solder joint external image set and the solder joint degradation features in the solder joint degradation state set to obtain a solder joint feature vector and a solder joint degradation feature vector;

[0039] Performing a feature alignment operation and an element-by-element fusion operation on the solder joint feature vector and the solder joint degradation feature vector in sequence to obtain an initial fused feature vector;

[0040] Using a statistical learning method to select a fused feature vector from the initial fused feature vector;

[0041] The integrated loss function is optimized using the fused feature vector until the solder joint degradation classification model converges.

[0042] Optionally, the step of acquiring a target solder joint image of a solder joint to be tested in the device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model comprises the following steps:

[0043] Acquire a target solder joint image of a solder joint to be tested in the device to be tested;

[0044] Preprocessing the target solder joint image;

[0045] Inputting the preprocessed target solder joint image into the trained solder joint feature detection model to obtain the target solder joint features output by the solder joint feature detection model;

[0046] The target solder joint feature is input into the trained solder joint state classification model to obtain a solder joint state classification result output by the solder joint state classification model.

[0047] In a second aspect, the present invention also provides a solder joint degradation state analysis system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0048] In a third aspect, the present invention further provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0049] The beneficial effects of the present invention are:

[0050] The deep learning-based solder joint degradation state analysis method adopted by the present invention includes the following steps: collecting device information and solder joint external images of multiple sample devices, and constructing a solder joint external image set in combination with the device information and the solder joint external images; processing the solder joint positions of multiple sample devices by a solder joint slicing method, and collecting solder joint internal images of all sample devices; constructing a solder joint degradation state set according to the solder joint internal images; constructing a solder joint degradation classification model based on a neural network model; training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set; obtaining a target solder joint image of a solder joint to be tested in the device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model; and determining the solder joint degradation state of the solder joint to be tested according to the solder joint state classification result.

[0051] Through the above steps, the external image of the solder joint of the sample device is first collected, and the solder joint features in the external image of the solder joint are annotated with an annotation tool to form a set of external images of the solder joint containing the solder joint features. Then, the solder joint of the sample device is sliced ​​to obtain the solder joint slicing results, and further a set of degradation states of the solder joint is formed. The classification model is trained with the external image set of the solder joint and the set of degradation states of the solder joint. The new solder joint image is input into the trained classification model to identify the degradation state classification result in the solder joint image, and then the degradation state of the solder joint is judged according to the degradation state classification result. Compared with manual visual inspection or direct slicing judgment, it will have higher detection efficiency and will not cause permanent damage to the device to be inspected. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the solder joint degradation state analysis method based on deep learning in the present invention. DETAILED DESCRIPTION

[0053] The invention discloses a solder joint degradation state analysis method based on deep learning.

[0054] Reference Figure 1 ,The solder point degradation state analysis method based on deep learning specifically includes the following steps:

[0055] S101. Collect device information and solder joint external images of multiple sample devices, and construct a solder joint external image set by combining the device information and solder joint external images.

[0056] First, it is necessary to collect the device information of various sample devices, including device model, device parameters, device solder joint location, etc. Then, collect the external image of the solder joint of the sample device, which can be taken by using a camera or other image acquisition device. Associate the device information with the external image of the solder joint, and construct a set of external images of the solder joint. A database or file system can be used to store and manage these image data.

[0057] S102. Process the solder joint positions of all sample devices according to the solder joint position information by using the solder joint slicing method, and collect internal images of the solder joints of all sample devices.

[0058] Among them, the solder joint positions of various sample devices need to be processed by the solder joint slicing method. The solder joint slicing method is a method of cutting the solder joint out of the welding structure to obtain the internal image of the solder joint. The solder joint slicing can be performed using a cutting tool or other suitable methods. After slicing, the internal images of the solder joints of all sample devices can be collected, and can be photographed using a camera or other image acquisition device.

[0059] S103. Construct a solder joint degradation state set according to the internal image of the solder joint.

[0060] Among them, it is necessary to construct a set of degradation states of solder joints based on the internal images of solder joints. Image processing and analysis methods can be used to extract the features of the internal images of solder joints, and the degradation state of solder joints can be judged based on these features. Machine learning algorithms or other suitable methods can be used for feature extraction and state judgment. A total of 8 different feature categories are set, namely solder joints, bulge, haw, fissure, rift, cracking, halo, else, representing solder joints, bulges, dragon scales, small gaps, small cracks, large cracks, annular damage, and other damage, respectively.

[0061] S104. Construct a solder joint degradation classification model based on a neural network model.

[0062] Among them, it is necessary to build a solder joint degradation classification model based on a neural network model. You can choose a suitable neural network structure and algorithm, and train the neural network model to classify the solder joint degradation state based on the characteristics of the internal image of the solder joint as input. You can use deep learning frameworks such as TensorFlow or PyTorch to implement the construction and training of the neural network model.

[0063] S105. Train a solder joint degradation classification model using a solder joint external image set and a solder joint degradation state set.

[0064] Among them, it is necessary to use the external image set of the solder joint and the degradation state set of the solder joint to train the solder joint degradation classification model. The external image of the solder joint is used as input, the degradation state of the solder joint is used as a label, and the training data is used to train the neural network model. Optimization algorithms such as batch gradient descent can be used to train the model and update parameters.

[0065] S106. Obtain a target solder joint image of the solder joint to be tested in the device to be tested, input the target solder joint image into the trained solder joint degradation classification model, and obtain a solder joint state classification result output by the solder joint degradation classification model.

[0066] Among them, it is necessary to obtain the target solder joint image of the solder joint to be tested in the device to be tested. The target solder joint image can be obtained by using a camera or other image acquisition device. Then, the target solder joint image is input into the trained solder joint degradation classification model, and the solder joint state classification result output by the solder joint degradation classification model is obtained through the forward propagation process of the model.

[0067] S107. Determine the solder joint degradation state of the solder joint to be tested according to the solder joint state classification result.

[0068] Among them, the degradation state of the solder joint to be tested is determined according to the solder joint state classification result. According to the classification result, the degree of degradation of the solder joint can be judged, such as healthy, slightly degraded, severely degraded, completely failed, etc. Different degradation state classification standards and judgment methods can be set according to specific needs and application scenarios. If there is no damage inside the solder joint, the solder joint is in a healthy state; when the crack inside the solder joint is small, the solder joint is marked as slightly degraded; when the crack inside the solder joint is large, the solder joint is marked as severely degraded; when the crack runs through the entire solder joint, the solder joint is judged to be completely failed.

[0069] The implementation principle of this embodiment is:

[0070] First, the external image of the solder joint of the sample device is collected, and the solder joint features in the external image of the solder joint are annotated with an annotation tool to form a set of external images of the solder joint containing the solder joint features. Then, the solder joint of the sample device is sliced ​​to obtain the solder joint slicing results, and further a set of solder joint degradation states is formed. The classification model is trained with the external image set of the solder joint and the set of solder joint degradation states. The new solder joint image is input into the trained classification model to identify the degradation state classification results in the solder joint image, and then the degradation state of the solder joint is judged according to the degradation state classification results. Compared with manual visual inspection or direct slicing judgment, it will have higher detection efficiency and will not cause permanent damage to the equipment to be inspected.

[0071] In one embodiment, step S102, that is, processing the solder joint positions of multiple sample devices by solder joint slicing method and collecting internal images of solder joints of all sample devices, specifically includes the following steps:

[0072] According to the solder point position information, the solder point positions of the various sample devices are cut into solder point slices according to preset sizes;

[0073] Inlay the solder joint slices into a preset standard template to form a solder joint standard template;

[0074] Grind and polish the standard sample of the welding point;

[0075] The image of the solder joint standard sample after grinding and polishing is collected by the solder joint image acquisition device as the internal image of the solder joint.

[0076] In this embodiment, first, the position of the weld is marked on the sample device according to the preset size and weld position information. Use appropriate cutting tools, such as a cutting machine, a laser cutter, etc., to cut out the weld position and maintain the consistency of size. Prepare a preset standard template, which can be a metal plate or other suitable material. Accurately inlay the weld slices onto the standard template to ensure that the position and size of the weld are consistent with the requirements of the standard template. Use appropriate fixing methods, such as welding, bonding, etc., to fix the weld slices on the standard template. Use sandpaper, grinding wheels or other grinding tools to grind the weld standard template.

[0077] Different grinding grain sizes and grinding methods can be selected as needed to gradually make the surface of the solder joint standard sample flat and smooth. During the grinding process, it is necessary to control the grinding force and direction to avoid damaging the solder joint standard sample. Use solder joint image acquisition equipment, such as cameras, microscopes, etc., to capture images of the solder joint standard sample after grinding and polishing. Ensure that the settings and parameters of the acquisition equipment are appropriate to obtain clear and accurate images of the inside of the solder joint. Perform appropriate lighting control and image processing on the solder joint standard sample to improve image quality and reduce noise.

[0078] In one embodiment, step S101, i.e. collecting device information and external images of solder joints of multiple sample devices, and building a solder joint external image set by combining the device information and the external images of solder joints, specifically includes the following steps:

[0079] Collect device information and external images of solder joints of various sample devices;

[0080] Extract solder joint process information of each sample device based on all device information;

[0081] Retrieving a corresponding solder joint feature set from a preset solder joint feature database based on the solder joint position information and the solder joint process information;

[0082] Bind each solder joint feature in the solder joint feature set to the corresponding solder joint external image one by one;

[0083] A weld external image set is constructed by combining all weld external images bound to weld features.

[0084] In this embodiment, appropriate equipment information collection tools, such as sensors, measuring instruments, etc., are used to collect equipment information of various sample equipment, including equipment model, equipment parameters, solder joint location information, etc. A camera or other image acquisition device is used to capture images of the exterior of solder joints of various sample equipment. Ensure that the image clearly and accurately captures the external features of the solder joints. Process the collected equipment information to extract solder joint process information of each sample equipment. Preset keyword extraction rules can be used to extract this information. The solder joint process information mainly reflects the welding method of the corresponding equipment.

[0085] Prepare a preset solder joint feature database, which contains the feature information of each solder joint, such as solder joint image features, solder joint year features, and solder joint process features. According to the solder joint location information and solder joint process information of the sample equipment, retrieve the corresponding solder joint feature set from the database. Match and bind the solder joint feature set retrieved from the database with the collected solder joint external image. You can use the device ID or other unique identifier to associate the solder joint feature and the solder joint external image. Ensure that each solder joint feature is correctly bound to the corresponding solder joint external image. Collect all solder joint external images bound with solder joint features together to form a solder joint external image set. You can use a database or file system to store and manage these solder joint external image data sets.

[0086] In one embodiment, step S103, i.e., constructing a solder joint degradation state set according to the internal image of the solder joint, specifically comprises the following steps:

[0087] Preprocessing the internal image of the solder joint to obtain a preprocessed solder joint image, the preprocessing step including an image enhancement step and an image rotation step;

[0088] The image processing method is used to extract the solder joint degradation features in the pre-processed solder joint image;

[0089] Marking degradation status of the internal image of the solder joint according to the degradation characteristics of the solder joint;

[0090] The degradation state set of solder joints is constructed by combining all the internal images of solder joints marked with degradation states.

[0091] In this embodiment, due to the influence of different operating environments and different processes, the degradation characteristics of solder joints are varied. However, the number of data sets is limited. While including more features, it is difficult to ensure that each type of feature has enough samples to support model learning and training. Data enhancement can effectively solve this problem; considering that the degradation damage of through-hole solder joints may occur at various positions of the solder joints, and in the process of collecting damage images, the samples can be rotated to collect the appearance of the damage at different angles. In this embodiment, data enhancement is achieved by rotating the pictures.

[0092] In one embodiment, step S103, i.e., constructing a solder joint degradation state set according to the internal image of the solder joint, specifically comprises the following steps:

[0093] Preprocessing the internal image of the solder joint to obtain a preprocessed solder joint image;

[0094] The basic solder joint degradation features in the pre-processed solder joint images are extracted using image processing methods;

[0095] Perform feature selection and feature dimension reduction on the basic solder joint degradation features to obtain the standard solder joint degradation features;

[0096] Marking degradation status for the internal image of the solder joint according to standard solder joint degradation characteristics;

[0097] The degradation state set of solder joints is constructed by combining all the internal images of solder joints marked with degradation states.

[0098] In this embodiment, the collected internal image of the solder joint is preprocessed, such as denoising, contrast enhancement, brightness adjustment, etc. The preprocessing step can be implemented using image processing software or an image processing library in a programming language. Use image processing algorithms or computer vision technology to extract basic solder joint degradation features in the preprocessed solder joint image. These features may include information such as the shape, color, texture, etc. of the solder joint, and feature extraction methods such as edge detection, color histogram, texture feature extraction, etc. can be used. Feature selection is performed on the extracted basic solder joint degradation features to select features that are useful for distinguishing the degradation state. Feature selection algorithms such as variance selection, correlation coefficient selection, chi-square test, etc. can be used for feature selection. For high-dimensional features, feature dimensionality reduction methods such as principal component analysis (PCA) and linear discriminant analysis (LDA) can be used to reduce the dimension of the features.

[0099] The degradation features of solder joints mainly include solder joints, bulges, dragon scales, small gaps, small cracks, large cracks, annular damage, and other damage. According to the size and position of the degradation features of solder joints, the severity of each degradation feature is quantitatively calculated. The severity of degradation features such as bulges, dragon scales, annular damage, and other damage is related to their area; the severity of degradation features such as small gaps, small cracks, and large cracks is related to their distance from the center of the solder joint and the cosine angle formed with the center of the solder joint. Based on the preprocessed solder joint images and the degradation features of solder joints, a classification algorithm or a regression algorithm is used to mark the degradation state of the internal image of the solder joint. Supervised learning algorithms such as support vector machines (SVM) and random forests can be used for classification or regression tasks. All internal images of solder joints marked with degradation states are collected together to form a set of degradation states of solder joints. Databases or file systems can be used to store and manage these solder joint degradation state data sets.

[0100] In one embodiment, step S104, i.e. building a solder joint degradation classification model based on a neural network model, specifically includes the following steps:

[0101] Based on the neural network model, a solder joint feature detection model and a solder joint status classification model are constructed respectively;

[0102] The output layer of the solder joint feature detection model is used as the input layer of the solder joint status classification model;

[0103] The loss functions of the solder joint feature detection model and the solder joint state classification model are combined to obtain a comprehensive loss function, and the solder joint feature detection model and the solder joint state classification model are combined into a solder joint degradation classification model.

[0104] In this embodiment, for the solder joint feature detection model, a convolutional neural network (CNN) or other suitable model structures may be used to design a network architecture and perform training to extract solder joint features from solder joint images. For the solder joint state classification model, a fully connected neural network, a support vector machine (SVM) or other suitable model structures may be used to design a network architecture and perform training to classify the degradation state of the solder joint.

[0105] The output of the feature extraction layer or the fully connected layer obtained by training the solder joint feature detection model is used as the input layer of the solder joint state classification model. Ensure that the input and output dimensions of the two models match for subsequent model combination and training. Define loss functions for the solder joint feature detection model and the solder joint state classification model, such as the cross entropy loss function, the mean square error loss function, etc. The loss functions of the two models are combined to obtain a comprehensive loss function by weighted summation or other methods. The comprehensive loss function can comprehensively consider the accuracy and consistency of solder joint feature detection and solder joint state classification. Combine the solder joint feature detection model and the solder joint state classification model to form an overall solder joint degradation classification model. The solder joint feature detection model is responsible for extracting the features of the solder joint image, while the solder joint state classification model classifies the degradation state based on these features. The two models can be combined into a solder joint degradation classification model by connecting the network layers of the two models, or by cascading or parallelizing.

[0106] In one embodiment, step S105, i.e., using the solder joint external image set and the solder joint degradation state set to train the solder joint degradation classification model, specifically includes the following steps:

[0107] The solder joint degradation features and the solder joint degradation state set in the solder joint external image set are converted into a structured data set, and each of the solder joint degradation features in the structured data set has a corresponding feature value;

[0108] Establishing the relationship between the degradation characteristics of the solder joints and the degradation state in the structured data set by using a support vector machine;

[0109] The structured data set is used to perform quantitative analysis and training on the solder joint degradation classification model to optimize the loss function of the solder joint degradation classification model, where the loss function includes positioning loss, confidence loss and classification loss.

[0110] In this embodiment, the solder joint degradation classification model can be built based on YOLOv5s, and YOLOv5s (hereinafter referred to as YOLOv5) has the advantages of fast training speed and strong real-time performance. The network structure of YOLOv5 is divided into four parts: input end, Backbone, Neck, and output end. The input end of YOLOv5 includes three methods: Mosaic data enhancement technology, image size scaling, and adaptive anchor frame calculation. Four pictures are randomly selected by Mosaic data enhancement technology for random scaling, cropping, and other operations, and then spliced ​​into one picture, enriching the training data set. The picture size scaling technology is used to scale the picture to a fixed size to improve the training speed. The optimal anchor frame value is generated by the adaptive anchor frame calculation method. Two CSP structures are used in YOLOv5. The backbone network Backbone uses the CSP1_X structure, and the Neck network uses the CSP2_X structure. X represents the number of residual components. The two CSP structures correspond to two networks, so that the output of the residual structure is easier to merge. In the post-processing process of target detection, the NMS method is usually used to screen multiple target boxes to enhance the detection ability of occluded targets.

[0111] The loss function is used to measure the difference between the model's predicted value and the true value, which largely determines the performance of the model. The three loss functions of YOLOv5 are classification loss clc_loss, positioning loss box_loss, and confidence loss obj_loss. The definition of the loss function is as follows:

[0112] Positioning loss: Bounding box prediction is one of the most important tasks in target detection. Yolov5 often uses CIoU as its positioning loss, which enables the center point and length of the real box and the predicted box to converge faster.

[0113] Confidence loss: The confidence of each prediction box indicates the reliability of the prediction box. The larger the value, the more reliable the prediction box is, and the closer it is to the true box. The confidence loss is obtained by calculating the second-level cross entropy for all prediction boxes.

[0114] Classification loss: Yolov5 uses the binary cross entropy function by default to calculate the classification loss.

[0115] In order to facilitate the learning of the classification model, the degradation feature set and the degradation state set need to be converted into a structured data set first; this data set gives a corresponding feature value for each degradation feature, and quantitatively analyzes different damage types by calculating the damage size and location of each solder joint image. The size of its feature value is determined by comparing the area of ​​the identified bulge (dragon scale) with the area of ​​the identified solder joint; for degradation features such as small gaps, small cracks, and large cracks, the ratio of its area to the area of ​​the solder joint is used, plus the angle formed by the feature surrounding the center of the solder joint, and the sum of these two values ​​is used as the feature value of its feature; for annular damage and other other damages, the distance between the area of ​​the identified damage and the relative center point is used as the basis for determining the size of its feature value. The relationship between degradation features and degradation states is learned by using SVM with RBF as the kernel function, and finally the solder joints are divided into four degradation states according to the degradation features: healthy state, mild degradation, severe degradation, and complete failure.

[0116] In one embodiment, step S105, i.e., using the solder joint external image set and the solder joint degradation state set to train the solder joint degradation classification model, specifically includes the following steps:

[0117] Performing standardized feature processing on the solder joint features in the solder joint external image set and the solder joint degradation features in the solder joint degradation state set to obtain a solder joint feature vector and a solder joint degradation feature vector;

[0118] Performing feature alignment and element-by-element fusion operations on the solder joint feature vector and the solder joint degradation feature vector in sequence to obtain an initial fused feature vector;

[0119] A statistical learning method is used to select a fused feature vector from the initial fused feature vector;

[0120] The comprehensive loss function is optimized using the fused feature vector until the solder joint degradation classification model converges.

[0121] In this embodiment, the solder joint features in the solder joint external image set are standardized, such as normalized, standardized, etc., so that they have the same scale and distribution. At the same time, the solder joint degradation features in the solder joint degradation state set are standardized so that they have the same scale and distribution. In this way, it can be ensured that the solder joint feature vector and the solder joint degradation feature vector are comparable in value.

[0122] Perform feature alignment on the solder joint feature vector and the solder joint degradation feature vector so that they have the same dimension and arrangement order. Perform element-by-element fusion operations on the feature-aligned vectors, such as element-by-element addition and element-by-element multiplication, to obtain the initial fused feature vector. In this way, the solder joint features and solder joint degradation features can be fused, and the external features and internal degradation state of the solder joint can be comprehensively considered. Statistical learning methods such as feature selection algorithms, principal component analysis (PCA), linear discriminant analysis (LDA), etc. are used to filter out fused feature vectors that are useful for degradation state classification from the initial fused feature vectors. These methods can be screened based on indicators such as feature correlation and variance to select feature vectors with high discrimination for classification tasks.

[0123] The fused feature vector is introduced into the comprehensive loss function to optimize the solder joint degradation classification model. Optimization algorithms such as gradient descent can be used to optimize the model parameters by minimizing the comprehensive loss function until the model converges. In this way, the accuracy and performance of the solder joint degradation classification model can be improved by optimizing the fused feature vector.

[0124] In one embodiment, step S106 is to obtain a target solder joint image of a solder joint to be tested in the device to be tested, input the target solder joint image into a trained solder joint degradation classification model, and obtain a solder joint state classification result output by the solder joint degradation classification model, which specifically includes the following steps:

[0125] Acquire a target solder joint image of a solder joint to be tested in the device to be tested;

[0126] Preprocessing target solder joint image;

[0127] Inputting the preprocessed target solder joint image into the trained solder joint feature detection model to obtain the target solder joint features output by the solder joint feature detection model;

[0128] The target solder joint features are input into the trained solder joint state classification model to obtain the solder joint state classification results output by the solder joint state classification model.

[0129] In this embodiment, image data of the solder joint to be tested is obtained from the device to be tested. It is ensured that the acquired image has sufficient resolution and quality so that the solder joint features can be accurately extracted and the state classification can be performed. The target solder joint image is preprocessed to improve the image quality and reduce the influence of noise. The preprocessing operation may include image denoising, image enhancement, image smoothing, etc., and an appropriate preprocessing method is selected according to the specific situation.

[0130] Input the preprocessed target solder joint image into the trained solder joint feature detection model. The solder joint feature detection model will perform feature extraction on the image and extract the key features of the solder joint. The output of the model can be a feature vector representing the features of the target solder joint. Input the solder joint features into the trained solder joint status classification model. The solder joint status classification model will classify the status of the solder joint based on the input features, such as healthy, slightly degraded, severely degraded, completely failed, etc. The output of the model can be a classification result representing the status of the target solder joint.

[0131] The present invention also discloses a deep learning-based solder joint degradation state analysis system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the deep learning-based solder joint degradation state analysis method in any of the above-mentioned embodiments is implemented.

[0132] The implementation principle of this embodiment is:

[0133] The following steps are performed by calling the program: collecting device information and external images of solder joints of multiple sample devices, and constructing a solder joint external image set in combination with the device information and the external images of solder joints; processing solder joint positions of multiple sample devices by a solder joint slicing method, and collecting internal images of solder joints of all sample devices; constructing a solder joint degradation state set according to the internal images of solder joints; constructing a solder joint degradation classification model based on a neural network model; training the solder joint degradation classification model using the external image set of solder joints and the solder joint degradation state set; obtaining a target solder joint image of a solder joint to be tested in the device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model; and determining the solder joint degradation state of the solder joint to be tested according to the solder joint state classification result.

[0134] Through the above steps, the external image of the solder joint of the sample device is first collected, and the solder joint features in the external image of the solder joint are annotated with an annotation tool to form a set of external images of the solder joint containing the solder joint features. Then, the solder joint of the sample device is sliced ​​to obtain the solder joint slicing results, and further a set of degradation states of the solder joint is formed. The classification model is trained with the external image set of the solder joint and the set of degradation states of the solder joint. The new solder joint image is input into the trained classification model to identify the degradation state classification result in the solder joint image, and then the degradation state of the solder joint is judged according to the degradation state classification result. Compared with manual visual inspection or direct slicing judgment, it will have higher detection efficiency and will not cause permanent damage to the device to be inspected.

[0135] The present invention also discloses a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the deep learning-based solder joint degradation state analysis method in any of the above-mentioned embodiments are implemented.

[0136] The implementation principle of this embodiment is:

[0137] The following steps are performed by calling the program: collecting device information and external images of solder joints of multiple sample devices, and constructing a solder joint external image set in combination with the device information and the external images of solder joints; processing solder joint positions of multiple sample devices by a solder joint slicing method, and collecting internal images of solder joints of all sample devices; constructing a solder joint degradation state set according to the internal images of solder joints; constructing a solder joint degradation classification model based on a neural network model; training the solder joint degradation classification model using the external image set of solder joints and the solder joint degradation state set; obtaining a target solder joint image of a solder joint to be tested in the device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model; and determining the solder joint degradation state of the solder joint to be tested according to the solder joint state classification result.

[0138] Through the above steps, the external image of the solder joint of the sample device is first collected, and the solder joint features in the external image of the solder joint are annotated with an annotation tool to form a set of external images of the solder joint containing the solder joint features. Then, the solder joint of the sample device is sliced ​​to obtain the solder joint slicing results, and further a set of degradation states of the solder joint is formed. The classification model is trained with the external image set of the solder joint and the set of degradation states of the solder joint. The new solder joint image is input into the trained classification model to identify the degradation state classification result in the solder joint image, and then the degradation state of the solder joint is judged according to the degradation state classification result. Compared with manual visual inspection or direct slicing judgment, it will have higher detection efficiency and will not cause permanent damage to the device to be inspected.

[0139] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as above, which are not provided in detail for the sake of simplicity.

[0140] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for analyzing solder joint degradation state based on deep learning, characterized in that: The steps include: Collecting device information and external images of solder joints of multiple sample devices, and building a set of external images of solder joints by combining the device information and the external images of solder joints, wherein the device information includes solder joint position information; Processing the solder joint positions of all the sample devices according to the solder joint position information and using a solder joint slicing method to collect internal images of the solder joints of all the sample devices; constructing a solder joint degradation state set according to the internal image of the solder joint; Constructing solder joint degradation classification model based on neural network model; Training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set; Acquire a target solder joint image of a solder joint to be tested in the device to be tested, input the target solder joint image into the trained solder joint degradation classification model, and obtain a solder joint state classification result output by the solder joint degradation classification model; The solder joint degradation state of the solder joint to be tested is determined according to the solder joint state classification result.

2. The method for analyzing solder joint degradation state based on deep learning according to claim 1, characterized in that: The steps of processing the solder joint positions of all the sample devices according to the solder joint position information and using the solder joint slicing method to collect the internal images of the solder joints of all the sample devices include the following steps: According to the solder point position information, the solder point positions of the plurality of sample devices are cut into solder point slices according to preset sizes; Inlaying the solder joint slices into a preset standard template to form a solder joint standard template; Grinding and polishing the solder joint standard sample; The image of the solder joint standard sample after grinding and polishing is collected by a solder joint image collection device as the internal image of the solder joint.

3. The method for analyzing solder joint degradation state based on deep learning according to claim 1, characterized in that: The collecting of device information and solder joint external images of multiple sample devices and combining the device information and the solder joint external images to construct a solder joint external image set comprises the following steps: Collect device information and external images of solder joints of various sample devices; Extracting solder joint process information of each of the sample devices according to all the device information; Retrieving a corresponding solder point feature set from a preset solder point feature database based on the solder point position information and the solder point process information; Binding each solder joint feature in the solder joint feature set to the corresponding solder joint external image one by one; A weld point external image set is constructed by combining all the weld point external images bound to the weld point features.

4. The method for analyzing solder joint degradation state based on deep learning according to claim 3, characterized in that: The step of constructing a solder joint degradation state set according to the internal image of the solder joint comprises the following steps: Preprocessing the internal image of the solder joint to obtain a preprocessed solder joint image, wherein the preprocessing step includes an image enhancement step and an image rotation step; Extracting solder joint degradation features from the preprocessed solder joint image using an image processing method; Marking a degradation state for the internal image of the solder joint according to the solder joint degradation feature; A solder joint degradation state set is constructed by combining all the solder joint internal images marked with the degradation state.

5. The method for analyzing solder joint degradation state based on deep learning according to claim 4, characterized in that: The construction of the solder joint degradation classification model based on the neural network model comprises the following steps: Based on the neural network model, a solder joint feature detection model and a solder joint status classification model are constructed respectively; Using the output layer of the solder joint feature detection model as the input layer of the solder joint state classification model; The loss functions of the solder joint feature detection model and the solder joint state classification model are combined to obtain a comprehensive loss function, and the solder joint feature detection model and the solder joint state classification model are combined into a solder joint degradation classification model.

6. The method for analyzing solder joint degradation state based on deep learning according to claim 5, characterized in that: The step of training the solder joint degradation classification model by using the solder joint external image set and the solder joint degradation state set comprises the following steps: The solder joint degradation features and the solder joint degradation state set in the solder joint external image set are converted into a structured data set, and each of the solder joint degradation features in the structured data set has a corresponding feature value; Establishing the relationship between the degradation characteristics of the solder joints and the degradation state in the structured data set by using a support vector machine; The structured data set is used to perform quantitative analysis and training on the solder joint degradation classification model to optimize the loss function of the solder joint degradation classification model, where the loss function includes positioning loss, confidence loss and classification loss.

7. The method for analyzing solder joint degradation state based on deep learning according to claim 5, characterized in that: The step of training the solder joint degradation classification model by using the solder joint external image set and the solder joint degradation state set comprises the following steps: Performing standardized feature processing on the solder joint features in the solder joint external image set and the solder joint degradation features in the solder joint degradation state set to obtain a solder joint feature vector and a solder joint degradation feature vector; Performing a feature alignment operation and an element-by-element fusion operation on the solder joint feature vector and the solder joint degradation feature vector in sequence to obtain an initial fused feature vector; Using a statistical learning method to select a fused feature vector from the initial fused feature vector; The integrated loss function is optimized using the fused feature vector until the solder joint degradation classification model converges.

8. The method for analyzing solder joint degradation state based on deep learning according to claim 6 or 7, characterized in that: The step of acquiring a target solder joint image of a solder joint to be tested in the device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model comprises the following steps: Acquire a target solder joint image of a solder joint to be tested in the device to be tested; Preprocessing the target solder joint image; Inputting the preprocessed target solder joint image into the trained solder joint feature detection model to obtain the target solder joint features output by the solder joint feature detection model; The target solder joint feature is input into the trained solder joint state classification model to obtain a solder joint state classification result output by the solder joint state classification model.

9. A solder joint degradation state analysis system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images