Deep learning-based solder joint degradation state analysis method and system, and storage medium

Abstract

The application provides a kind of based on deep learning's welding point degradation state analysis method, system and storage medium, the method includes the following steps: the equipment information and welding point external image of multiple sample equipment are collected, and welding point external image set is constructed in combination with equipment information and welding point external image;According to the welding point position information and by welding point slicing method processing welding point position, the welding point internal image of all sample equipment is collected;According to welding point internal image, welding point degradation state set is constructed;Based on neural network model, welding point degradation classification model is constructed;Welding point degradation classification model is trained using welding point external image set and welding point degradation state set;Target welding point image is input into the welding point degradation classification model after training, and the welding point state classification result output by welding point degradation classification model is obtained;According to welding point state classification result, determine the welding point degradation state of the welding point to be measured.The application has the effect that when detecting the welding point degradation of equipment, the detection efficiency is higher and the equipment damage is not caused.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of device solder joint analysis, and particularly relates to a solder joint degradation state analysis method and system based on deep learning and a storage medium. BACKGROUND

[0002] Electronic packaging technology is developing rapidly, and components are becoming smaller and more integrated, with internal solder joints being even smaller in size and more in number. The main function of internal solder joints is mechanical support and electrical connection. Once a solder joint is broken, it will cause the entire component to malfunction or fail, resulting in serious consequences. If the degradation state of each electronic board card can be evaluated during maintenance, it can be determined whether the board card needs to be replaced in advance, which can greatly reduce the risk of equipment failure and improve the reliability of electronic board cards.

[0003] Traditional methods for evaluating the degradation state of solder joints include manual visual inspection and slicing, as well as electrical parameter and strain parameter testing. However, electrical testing is not suitable for existing maintenance board cards, strain parameters are difficult to obtain, and the change characteristics are not obvious. Manual visual inspection requires high skill from maintenance personnel and does not meet the needs of large-scale maintenance, with low detection efficiency. Moreover, slicing detection is destructive, and the electronic board card cannot be used after slicing. SUMMARY

[0004] The application provides a solder joint degradation state analysis method and system based on deep learning and a storage medium to solve the problem of low detection efficiency and damage to equipment during detection.

[0005] In a first aspect, the application provides a solder joint degradation state analysis method based on deep learning, which includes the following steps:

[0006] Collecting device information and solder joint external images of multiple sample devices, and constructing a solder joint external image set in combination with the device information and the solder joint external images, wherein the device information includes solder joint position information;

[0007] According to the solder joint position information and by solder joint slicing method, processing the solder joint positions of all the sample devices to collect solder joint internal images of all the sample devices;

[0008] Constructing a solder joint degradation state set according to the solder joint internal images;

[0009] Constructing a solder joint degradation classification model based on a neural network model;

[0010] Training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set;

[0011] Obtaining a target solder joint image of a solder joint to be tested in a device to be tested, inputting the target solder joint image into the trained solder joint degradation classification model to obtain a solder joint state classification result output by the solder joint degradation classification model;

[0012] According to the solder joint state classification result, the solder joint degradation state of the solder joint to be tested is determined.

[0013] Optionally, the solder joint position of all the sample devices is processed according to the solder joint position information and by a solder joint slicing method, and the solder joint internal images of all the sample devices are collected, including the following steps:

[0014] According to the solder joint position information, the solder joint positions of a plurality of sample devices are cut into solder joint slices according to a preset size;

[0015] Embedding the solder joint slices into a preset standard template to form a solder joint standard template;

[0016] Grinding and polishing the solder joint standard template;

[0017] Collecting the image of the solder joint standard template after grinding and polishing as a solder joint internal image by a solder joint image collection device.

[0018] Optionally, the device information and the solder joint external image of a plurality of sample devices are collected, and a solder joint external image set is constructed in combination with the device information and the solder joint external image, including the following steps:

[0019] Collecting the device information and the solder joint external image of a plurality of sample devices;

[0020] Extracting the solder joint process information of each sample device according to all the device information;

[0021] Based on the solder joint position information and the solder joint process information, corresponding solder joint feature sets are retrieved from a preset solder joint feature database;

[0022] Each solder joint feature in the solder joint feature set is bound to the corresponding solder joint external image one by one;

[0023] Combining all the solder joint external images bound with the solder joint features to construct a solder joint external image set.

[0024] Optionally, the solder joint degradation state set is constructed according to the solder joint internal image, including the following steps:

[0025] Preprocessing the solder joint internal image to obtain a preprocessed solder joint image, the preprocessing step including an image enhancement step and an image rotation step;

[0026] Using an image processing method to extract solder joint degradation features in the preprocessed solder joint image;

[0027] According to the welding point degradation feature, the internal image of the welding point is marked with a degradation state;

[0028] The internal images of the welding points marked with the degradation state are combined to construct a welding point degradation state set.

[0029] Optionally, the construction of the welding point degradation classification model based on the neural network model comprises the following steps:

[0030] The welding point feature detection model and the welding point state classification model are constructed based on the neural network model respectively;

[0031] The output layer of the welding point feature detection model is used as the input layer of the welding point state classification model;

[0032] The loss functions of the welding point feature detection model and the welding point state classification model are combined to obtain a comprehensive loss function, and the welding point feature detection model and the welding point state classification model are combined into a welding point degradation classification model.

[0033] Optionally, the training of the welding point degradation classification model by using the welding point external image set and the welding point degradation state set comprises the following steps:

[0034] The welding point features in the welding point external image set and the welding point degradation state set are converted into a structured data set, and each welding point feature in the structured data set has a corresponding feature value;

[0035] The relationship between the welding point features and the degradation states in the structured data set is established by using a support vector machine;

[0036] The welding point degradation classification model is quantitatively analyzed and trained by using the structured data set, so as to optimize the loss function of the welding point degradation classification model, and the loss function comprises a positioning loss, a confidence loss and a classification loss.

[0037] Optionally, the training of the welding point degradation classification model by using the welding point external image set and the welding point degradation state set comprises the following steps:

[0038] The welding point features in the welding point external image set and the welding point degradation features in the welding point degradation state set are standardized and processed to obtain welding point feature vectors and welding point degradation feature vectors;

[0039] The welding point feature vectors and the welding point degradation feature vectors are sequentially subjected to feature alignment operations and element-by-element fusion operations to obtain initial fusion feature vectors;

[0040] A statistical learning method is used to screen fusion feature vectors from the initial fusion feature vectors.

[0041] optimizing the comprehensive loss function by using the fusion feature vector until the welding point degradation classification model converges.

[0042] Optionally, the method comprises the following steps of: acquiring a target welding point image of a welding point to be tested in a device to be tested; inputting the target welding point image into the trained welding point degradation classification model to obtain a welding point state classification result output by the welding point degradation classification model.

[0043] acquiring a target welding point image of a welding point to be tested in a device to be tested;

[0044] preprocessing the target welding point image;

[0045] inputting the preprocessed target welding point image into the trained welding point feature detection model to obtain a target welding point feature output by the welding point feature detection model;

[0046] inputting the target welding point feature into the trained welding point state classification model to obtain a welding point state classification result output by the welding point state classification model.

[0047] In a second aspect, the present application further provides a welding point degradation state analysis system based on deep learning, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of the first aspect when executing the computer program.

[0048] In a third aspect, the present application further provides a computer storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the method of the first aspect.

[0049] The present application has the following advantages:

[0050] The welding point degradation state analysis method based on deep learning comprises the following steps: collecting device information and welding point external images of a plurality of sample devices, and constructing a welding point external image set in combination with the device information and the welding point external images; processing welding point positions of the plurality of sample devices by a welding point slicing method, and collecting welding point internal images of all the sample devices; constructing a welding point degradation state set according to the welding point internal images; constructing a welding point degradation classification model based on a neural network model; training the welding point degradation classification model by using the welding point external image set and the welding point degradation state set; acquiring a target welding point image of a welding point to be tested in a device to be tested, inputting the target welding point image into the trained welding point degradation classification model to obtain a welding point state classification result output by the welding point degradation classification model; and determining a welding point degradation state of the welding point to be tested according to the welding point state classification result.

[0051] Through the above steps, first, the solder joint external image of the sample device is collected, the solder joint features in the solder joint external image are labeled with a labeling tool to form a solder joint external image set containing solder joint features, then the solder joint of the sample device is sliced to obtain a solder joint slice result, and a solder joint degradation state set is further formed. The solder joint external image set and the solder joint degradation state set are used to train a classification model. The new solder joint image is input into the trained classification model to identify the degradation state classification result in the solder joint image, and then the solder joint degradation state is judged according to the degradation state classification result. Compared with artificial visual inspection or direct slicing judgment, the detection efficiency is higher, and the permanent damage to the device to be detected is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is a flowchart of the solder joint degradation state analysis method based on deep learning in the present application. DETAILED DESCRIPTION

[0053] The present application discloses a solder joint degradation state analysis method based on deep learning.

[0054] REFERENCE Figure 1 The solder joint degradation state analysis method based on deep learning specifically includes the following steps:

[0055] S101. Collect device information and solder joint external images of multiple sample devices, and construct a solder joint external image set in combination with the device information and the solder joint external images.

[0056] First, the device information of multiple sample devices needs to be collected, including device model, device parameters, device solder joint position, etc. Then, the solder joint external images of the sample devices are collected, which can be photographed using a camera or other image collection device. The device information and the solder joint external images are associated, and a solder joint external image set is constructed, which can use a database or a file system to store and manage these image data.

[0057] S102. According to the solder joint position information and by the solder joint slicing method, process the solder joint position of all sample devices to collect the solder joint internal images of all sample devices.

[0058] Among them, the solder joint position of multiple sample devices needs to be processed by the solder joint slicing method. The solder joint slicing method is a method of cutting the solder joint from the welding structure to obtain the solder joint internal image. Cutting tools or other suitable methods can be used for solder joint slicing. After slicing, the solder joint internal images of all sample devices can be collected, which can be photographed using a camera or other image collection device.

[0059] S103. Construct a solder joint degradation state set according to the solder joint internal images.

[0060] The solder joint degradation state set needs to be constructed based on the internal images of the solder joints. Image processing and analysis methods can be used to extract features of the internal images of the solder joints, and the degradation state of the solder joints can be determined based on these features. Machine learning algorithms or other suitable methods can be used for feature extraction and state determination. A total of 8 different feature categories are set, namely solder joints, bulge, haw, fissure, rift, cracking, halo, and else, representing solder joints, bulges, dragon scales, small cracks, small cracks, large cracks, ring-shaped damage, and other damage.

[0061] S104. Construct a solder joint degradation classification model based on a neural network model.

[0062] The solder joint degradation classification model needs to be constructed based on the neural network model. Suitable neural network structures and algorithms can be selected, and the features of the internal images of the solder joints can be used as input to train the neural network model to classify the solder joint degradation state. Deep learning frameworks such as TensorFlow or PyTorch can be used to implement the construction and training of the neural network model.

[0063] S105. Train the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set.

[0064] The solder joint degradation classification model needs to be trained using the solder joint external image set and the solder joint degradation state set. The solder joint external image is used as input, and the solder joint degradation state is used as label. The neural network model is trained using the training data. Batch gradient descent and other optimization algorithms can be used to train and update the parameters of the model.

[0065] S106. Obtain the target solder joint image of the solder joint to be tested in the device to be tested, input the target solder joint image into the trained solder joint degradation classification model, and obtain the solder joint state classification result output by the solder joint degradation classification model.

[0066] The target solder joint image of the solder joint to be tested in the device to be tested needs to be obtained. A camera or other image acquisition device can be used to obtain the target solder joint image. Then, the target solder joint image is input into the trained solder joint degradation classification model, and the solder joint state classification result output by the solder joint degradation classification model is obtained through the forward propagation process of the model.

[0067] S107. Determine the solder joint degradation state of the solder joint to be tested according to the solder joint state classification result.

[0068] The solder degradation state of the to-be-tested solder is determined according to the solder state classification result. According to the classification result, the degradation degree of the solder can be judged, such as healthy, slight degradation, severe degradation, and complete failure. Different degradation state classification standards and judgment methods can be set according to specific requirements and application scenarios. If there is no damage inside the solder, the solder is in a healthy state; when the internal crack of the solder is small, the solder is marked as slight degradation; when the internal crack of the solder is large, the solder is marked as severe degradation; and when the crack penetrates the entire solder, the solder is determined as complete failure.

[0069] The implementation principle of the embodiment is as follows:

[0070] First, the solder external image of the sample device is collected, the solder features in the solder external image are labeled by using a labeling tool to form a solder external image set containing solder features, the sample device is then subjected to solder slicing processing to obtain a solder slicing result, and a solder degradation state set is further formed. The solder external image set and the solder degradation state set are used to train a classification model. A new solder image is input into the trained classification model to identify the degradation state classification result of the solder image, and then the solder degradation state is determined according to the degradation state classification result. Compared with manual visual inspection or direct slicing judgment, the method has higher detection efficiency and does not cause permanent damage to the to-be-tested device.

[0071] In one embodiment, step S102 is to process the solder positions of a plurality of sample devices by a solder slicing method, and the solder internal images of all sample devices are collected, which specifically includes the following steps:

[0072] According to the solder position information, the solder positions of the plurality of sample devices are cut into solder slices according to a preset size;

[0073] The solder slices are inlaid into a preset standard template to form a solder standard template;

[0074] The solder standard template is subjected to grinding and polishing processing;

[0075] An image of the solder standard template after the grinding and polishing processing is collected by a solder image collection device as a solder internal image.

[0076] In this embodiment, first, according to the preset size and the welding point position information, the position of the welding point is marked on the sample device. Using appropriate cutting tools, such as cutting machines, laser cutting machines, etc., the welding point position is cut out and the size consistency is maintained. A preset standard sample plate can be prepared, which can be a metal plate or other suitable materials. The welding point slice is precisely inlaid on the standard sample plate, ensuring that the position and size of the welding point are consistent with the requirements of the standard sample plate. Using appropriate fixing methods, such as welding, bonding, etc., the welding point slice is fixed on the standard sample plate. Using sandpaper, grinding wheel or other grinding tools, the welding point standard sample plate is ground.

[0077] According to the needs, different grinding granularity and grinding methods can be selected to gradually smooth the surface of the welding point standard sample plate. During the grinding process, attention should be paid to control the grinding force and direction to avoid damaging the welding point standard sample plate. Using welding point image acquisition equipment, such as camera, microscope, etc., the image of the welding point standard sample plate after polishing is collected. Ensure that the settings and parameters of the acquisition equipment are appropriate to obtain clear and accurate internal images of the welding point. Proper lighting control and image processing are performed on the welding point standard sample plate to improve image quality and reduce noise.

[0078] In one embodiment, step S101 of collecting device information and welding point external image of a plurality of sample devices and constructing a welding point external image set in combination with the device information and welding point external image includes the following steps:

[0079] Collecting device information and welding point external image of a plurality of sample devices;

[0080] Extracting welding point process information of each sample device according to all device information;

[0081] Retrieving corresponding welding point feature set from the preset welding point feature database based on the welding point position information and the welding point process information;

[0082] Binding each welding point feature in the welding point feature set with the corresponding welding point external image one by one;

[0083] Constructing a welding point external image set in combination with all welding point external images bound with welding point features.

[0084] In this embodiment, using appropriate device information collection tools, such as sensors, measuring instruments, etc., the device information of a plurality of sample devices is collected, including device model, device parameters, welding point position information, etc. Using a camera or other image acquisition equipment, the external image of the welding point of a plurality of sample devices is collected. Ensure that the image clearly and accurately captures the external features of the welding point. Process the collected device information to extract the welding point process information of each sample device. The preset keyword extraction rule can be used to extract these information. The welding point process information mainly reflects the welding method of the corresponding device.

[0085] A preset weld feature database is prepared, which contains feature information of each weld, such as weld image features, weld age features, and weld process features, etc. According to the weld position information and weld process information of the sample equipment, the corresponding weld feature set is retrieved from the database. The weld feature set retrieved from the database is matched and bound with the collected weld external image. The device ID or other unique identifier can be used to associate the weld feature and the weld external image. Ensure that each weld feature is correctly bound with the corresponding weld external image. All weld external images bound with weld features are collected together to form a weld external image set. The database or file system can be used to store and manage these weld external image data sets.

[0086] In one embodiment, step S103 of constructing a weld degradation state set according to the weld internal image specifically includes the following steps:

[0087] The weld internal image is preprocessed to obtain a preprocessed weld image, and the preprocessing step includes an image enhancement step and an image rotation step.

[0088] Weld degradation features in the preprocessed weld image are extracted using image processing methods.

[0089] The weld internal image is labeled with a degradation state according to the weld degradation features.

[0090] All weld internal images labeled with a degradation state are combined to construct a weld degradation state set.

[0091] In this embodiment, due to the influence of different operating environments and different processes, the degradation features of the welds are diverse. However, the number of data sets is limited, and while containing more features, it is difficult to ensure that each type of feature has enough samples to support model learning and training. Data augmentation can effectively solve this problem; considering that the degradation damage of the through-hole weld may occur at various positions of the weld, and during the collection of the damage image, the appearance of the damage at different angles can be collected by rotating the sample, in this embodiment, data augmentation is achieved by rotating the picture.

[0092] In one embodiment, step S103 of constructing a weld degradation state set according to the weld internal image specifically includes the following steps:

[0093] The weld internal image is preprocessed to obtain a preprocessed weld image.

[0094] Basic weld degradation features in the preprocessed weld image are extracted using image processing methods.

[0095] The basic weld degradation features are subjected to feature selection and feature dimension reduction processing to obtain standard weld degradation features.

[0096] According to the standard solder degradation feature, the degradation state is marked for the solder internal image;

[0097] The solder degradation state set is constructed by combining all the solder internal images marked with the degradation state.

[0098] In this embodiment, the collected solder internal image is preprocessed, such as denoising, enhancing contrast, adjusting brightness, etc. The preprocessing step can be realized by using image processing software or image processing library in programming language. Using image processing algorithm or computer vision technology, the basic solder degradation features in the preprocessed solder image are extracted. These features can include shape, color, texture, etc. of the solder, and feature extraction methods such as edge detection, color histogram, texture feature extraction, etc. can be used. Feature selection is performed on the extracted basic solder degradation features, and features useful for degradation state discrimination are selected. Feature selection algorithms such as variance selection, correlation coefficient selection, chi-square test, etc. can be used for feature selection. For high-dimensional features, feature dimension reduction methods such as principal component analysis (PCA), linear discriminant analysis (LDA), etc. can be used to reduce the dimension of the features.

[0099] The solder degradation features mainly include solder, bulge, dragon scale, small gap, small crack, large crack, ring damage, and other damage. According to the size and position of the solder degradation features, the severity of each degradation feature is quantitatively calculated. The severity of the bulge, dragon scale, ring damage, and other damage is related to the area; the severity of the small gap, small crack, and large crack is related to the distance from the center of the solder and the cosine angle formed with the center of the solder. Based on the preprocessed solder image and the solder degradation feature, a classification algorithm or a regression algorithm is used to mark the degradation state for the solder internal image. Supervised learning algorithms such as support vector machine (SVM), random forest (Random Forest), etc. can be used for classification or regression tasks. All the solder internal images marked with the degradation state are collected together to form a solder degradation state set. Database or file system can be used to store and manage these solder degradation state data sets.

[0100] In one embodiment, step S104 of constructing a solder degradation classification model based on a neural network model specifically includes the following steps:

[0101] Based on the neural network model, a solder feature detection model and a solder state classification model are constructed respectively;

[0102] The output layer of the solder feature detection model is used as the input layer of the solder state classification model;

[0103] The loss functions of the solder joint feature detection model and the solder joint state classification model are combined to obtain a comprehensive loss function, and the solder joint feature detection model and the solder joint state classification model are combined into a solder joint degradation classification model.

[0104] In this embodiment, for the solder joint feature detection model, a convolutional neural network (CNN) or other suitable model structure can be used to design the network architecture and perform training to extract solder joint features from solder joint images. For the solder joint state classification model, a fully connected neural network, support vector machine (SVM), or other suitable model structure can be used to design the network architecture and perform training to classify the degradation state of the solder joint.

[0105] The output of the feature extraction layer or the fully connected layer obtained by training the solder joint feature detection model is used as the input layer of the solder joint state classification model. The input and output dimensions of the two models are matched to facilitate subsequent model combination and training. Loss functions such as cross-entropy loss function, mean square error loss function, etc. are defined for the solder joint feature detection model and the solder joint state classification model. The loss functions of the two models are combined, which can be obtained by weighted summation or other ways. The comprehensive loss function can comprehensively consider the accuracy and consistency of solder joint feature detection and solder joint state classification. The solder joint feature detection model and the solder joint state classification model are combined to form a whole solder joint degradation classification model. The solder joint feature detection model is responsible for extracting the features of the solder joint image, while the solder joint state classification model classifies the degradation state according to these features. The network layers of the two models can be connected, or the two models can be combined into a solder joint degradation classification model through cascading or parallel mode.

[0106] In one embodiment, step S105 of training the solder joint degradation classification model using the solder joint external image set and the solder joint degradation state set specifically includes the following steps:

[0107] The solder joint features in the solder joint external image set and the solder joint degradation state set are converted into a structured data set, and each solder joint feature in the structured data set has a corresponding feature value;

[0108] The relationship between the solder joint features and the degradation states in the structured data set is established by a support vector machine;

[0109] The structured data set is used to quantitatively analyze and train the solder joint degradation classification model to optimize the loss function of the solder joint degradation classification model, and the loss function includes a positioning loss, a confidence loss, and a classification loss.

[0110] In the embodiment, the solder joint degradation classification model can be constructed based on YOLOv5s, which has the advantages of fast training speed and strong real-time performance. The network structure of YOLOv5 includes four parts: input end, backbone, neck, and output end. The input end of YOLOv5 includes three methods: Mosaic data enhancement technology, picture size scaling, and adaptive anchor box calculation. Through the Mosaic data enhancement technology, four pictures are randomly selected and scaled, cropped, and spliced into one picture, enriching the training data set. Through the picture size scaling technology, the picture is scaled to a fixed size, improving the training speed. Through the adaptive anchor box calculation method, the best anchor box value is generated. Two CSP structures are used in YOLOv5. The backbone network uses CSP1_X structure, and the neck network uses CSP2_X structure. X in them represents the number of residual components. The two CSP structures correspond to two networks, so that the output of the residual structure is more easily fused. In the post-processing of target detection, the NMS method is usually used to filter multiple target boxes, and the detection ability of occluded targets is enhanced.

[0111] The loss function is used to measure the degree of difference between the model's predicted value and the true value, and to a great extent determines the performance of the model. The three loss functions of YOLOv5 are classification loss clc_loss, positioning loss box_loss, and confidence loss obj_loss. The definition of the loss function is as follows:

[0112] Positioning loss: bounding box prediction is one of the most important tasks in target detection, and yolov5 commonly uses CIoU as its positioning loss, which makes the center points and the size of the real box and the predicted box converge faster.

[0113] Confidence loss: the confidence of each predicted box represents the reliability of the predicted box. The greater the value, the more reliable the predicted box, and the closer it is to the real box. The confidence loss is obtained by calculating the cross-entropy of all predicted boxes.

[0114] Classification loss: Yolov5 uses the binary cross-entropy function to calculate the classification loss by default.

[0115] In order to facilitate the learning of the classification model, the degradation feature set and the degradation state set need to be first converted into a structured data set; the data set gives a corresponding feature value for each degradation feature, and through calculation of the damage size and the position of the damage of each solder joint image, quantitative analysis is performed on different damage types. By comparing the area of the identified bump (dragon scale) with the area of the identified solder joint, the size of the feature value is determined; for small gap, small crack, large crack and other degradation features, the ratio of the identified area to the area of the solder joint is used, and the angle formed by the feature surrounding the center of the solder joint is added, and the sum of the two values is used as the feature value of the feature; for ring damage and other damage, the area of the damage and the distance from the relative center point are used as the basis for determining the size of the feature value. The relationship between the degradation feature and the degradation state is learned by using the SVM with RBF as the kernel function, and finally the solder joint is divided into four degradation states of healthy state, mild degradation, severe degradation and complete failure according to the degradation feature.

[0116] In one embodiment, the step S105 of training the solder joint degradation classification model by using the solder joint external image set and the solder joint degradation state set comprises the following steps:

[0117] The solder joint features in the solder joint external image set and the solder joint degradation features in the solder joint degradation state set are subjected to standard feature processing to obtain solder joint feature vectors and solder joint degradation feature vectors;

[0118] The solder joint feature vectors and the solder joint degradation feature vectors are subjected to feature alignment operation and element-by-element fusion operation in sequence to obtain initial fusion feature vectors;

[0119] The statistical learning method is used to screen fusion feature vectors from the initial fusion feature vectors;

[0120] The fusion feature vectors are used to optimize the comprehensive loss function until the solder joint degradation classification model converges.

[0121] In this embodiment, the solder joint features in the solder joint external image set are subjected to standardization processing such as normalization and standardization, so as to have the same scale and distribution. At the same time, the solder joint degradation features in the solder joint degradation state set are subjected to standardization processing, so as to have the same scale and distribution. In this way, the solder joint feature vectors and the solder joint degradation feature vectors can have comparability in numerical value.

[0122] The feature alignment operation is performed on the welding point feature vector and the welding point degradation feature vector to make them have the same dimension and arrangement order. Element-wise fusion operations such as element-wise addition, element-wise multiplication, etc. are performed on the feature-aligned vectors to obtain an initial fusion feature vector. In this way, the welding point features and the welding point degradation features can be fused, and the external features and internal degradation states of the welding points can be considered comprehensively. Statistical learning methods such as feature selection algorithms, principal component analysis (PCA), linear discriminant analysis (LDA), etc. are used to select fusion feature vectors useful for classification of the degradation state from the initial fusion feature vector. These methods can select feature vectors with high discrimination for the classification task according to the correlation, variance, etc. of the features.

[0123] The fusion feature vector is introduced into the comprehensive loss function to optimize the welding point degradation classification model. Gradient descent and other optimization algorithms can be used to minimize the comprehensive loss function to optimize the model parameters until the model converges. In this way, the accuracy and performance of the welding point degradation classification model can be improved through the optimization of the fusion feature vector.

[0124] In one embodiment, the step S106 of obtaining the target welding point image of the welding point to be tested in the device to be tested, inputting the target welding point image into the trained welding point degradation classification model, and obtaining the welding point state classification result output by the welding point degradation classification model includes the following steps:

[0125] Obtain the target welding point image of the welding point to be tested in the device to be tested;

[0126] Preprocess the target welding point image;

[0127] Input the preprocessed target welding point image into the trained welding point feature detection model to obtain the target welding point feature output by the welding point feature detection model;

[0128] Input the target welding point feature into the trained welding point state classification model to obtain the welding point state classification result output by the welding point state classification model.

[0129] In this embodiment, image data of the welding point to be tested is obtained from the device to be tested. It is ensured that the obtained image has sufficient resolution and quality so that the welding point features can be accurately extracted and the state classification can be performed. The target welding point image is preprocessed to improve the image quality and reduce the influence of noise. The preprocessing operation can include image denoising, image enhancement, image smoothing, etc. and appropriate preprocessing methods are selected according to the specific situation.

[0130] The preprocessed target weld point image is input into the trained weld point feature detection model. The weld point feature detection model will perform feature extraction on the image, extracting the key features of the weld point. The output of the model can be a feature vector representing the features of the target weld point. The weld point features are input into the trained weld point state classification model. The weld point state classification model will classify the state of the weld point according to the input features, such as healthy, mild degradation, severe degradation, complete failure, etc. The output of the model can be a classification result representing the state of the target weld point.

[0131] The application also discloses a deep learning-based weld point degradation state analysis system, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the deep learning-based weld point degradation state analysis method in any of the above embodiments.

[0132] The implementation principle of the embodiment is as follows:

[0133] The following steps are performed by calling the program: collecting device information and weld point external images of a plurality of sample devices, and constructing a weld point external image set in combination with the device information and the weld point external images; processing the weld point positions of the plurality of sample devices by a weld point slicing method, and collecting weld point internal images of all the sample devices; constructing a weld point degradation state set according to the weld point internal images; constructing a weld point degradation classification model based on a neural network model; training the weld point degradation classification model using the weld point external image set and the weld point degradation state set; obtaining a target weld point image of a to-be-tested weld point in a to-be-tested device, inputting the target weld point image into the trained weld point degradation classification model, and obtaining a weld point state classification result output by the weld point degradation classification model; and determining a weld point degradation state of the to-be-tested weld point according to the weld point state classification result.

[0134] Through the above steps, first, the weld point external images of the sample devices are collected, the weld point features in the weld point external images are labeled with a labeling tool to form a weld point external image set containing weld point features, then the sample devices are subjected to weld point slicing processing to obtain weld point slicing results and further form a weld point degradation state set. The classification model is trained using the weld point external image set and the weld point degradation state set. A new weld point image is input into the trained classification model to identify the degradation state classification result in the weld point image, and then the weld point degradation state is determined according to the degradation state classification result. Compared with manual visual inspection or direct slicing judgment, the detection efficiency is higher, and the to-be-tested device is not permanently damaged.

[0135] The application also discloses a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the deep learning-based weld point degradation state analysis method in any of the above embodiments.

[0136] The implementation principle of the embodiment is:

[0137] The following steps are executed through the calling of the program: collecting device information and solder joint external images of a plurality of sample devices, and constructing a solder joint external image set in combination with the device information and the solder joint external images; processing solder joint positions of a plurality of sample devices through a solder joint slicing method, and collecting solder joint internal images of all sample devices; constructing a solder joint degradation state set according to the solder joint internal images; constructing a solder joint degradation classification model based on a neural network model; training the solder joint degradation classification model based on the solder joint external image set and the solder joint degradation state set; obtaining a target solder joint image of a to-be-tested solder joint in a to-be-tested device, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining a solder joint state classification result output by the solder joint degradation classification model; and determining a solder joint degradation state of the to-be-tested solder joint according to the solder joint state classification result.

[0138] Through the above steps, first, solder joint external images of sample devices are collected, solder joint features in the solder joint external images are labeled with a labeling tool to form a solder joint external image set containing solder joint features, then the sample devices are subjected to solder joint slicing processing to obtain solder joint slicing results and further form a solder joint degradation state set. The classification model is trained with the solder joint external image set and the solder joint degradation state set. A new solder joint image is input into the trained classification model to identify a degradation state classification result in the solder joint image, and then the solder joint degradation state is determined according to the degradation state classification result. Compared with artificial visual inspection or direct slicing judgment, the detection efficiency is higher, and the to-be-tested device is not permanently damaged.

[0139] It should be understood by those of ordinary skill in the art that the above discussion of any of the embodiments is merely exemplary and is not intended to suggest that the scope of protection of the present application is limited to these examples; under the idea of the present application, the above embodiments or technical features in different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

[0140] One or more embodiments of the present application are intended to cover all such alternatives, modifications and variations falling within the broad scope of the present application. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for analyzing the degradation state of solder joints based on deep learning, characterized in that, Includes the following steps: Collect equipment information and external images of solder joints from various sample devices, wherein the equipment information includes solder joint location information; Based on all the equipment information, extract the solder joint process information of each of the sample devices; Based on the solder joint location information and the solder joint process information, the corresponding solder joint feature set is retrieved from the preset solder joint feature database; Each solder joint feature in the solder joint feature set is bound to the corresponding external image of the solder joint; Construct a set of solder joint external images by combining all the external images of the solder joints that are bound to the solder joint features; Based on the solder joint location information, the solder joint locations of all the sample devices are processed using the solder joint slicing method, and internal images of the solder joints of all the sample devices are acquired. The image inside the solder joint is preprocessed to obtain a preprocessed solder joint image. The preprocessing steps include an image enhancement step and an image rotation step. Image processing methods are used to extract solder joint degradation features from the preprocessed solder joint image; The degradation state of the internal image of the solder joint is marked according to the degradation characteristics of the solder joint. A set of solder joint degradation states is constructed by combining all the internal images of the solder joints marked with the degradation state; Based on neural network models, a solder joint feature detection model and a solder joint status classification model are constructed respectively. The output layer of the solder joint feature detection model is used as the input layer of the solder joint state classification model; By combining the loss functions of the solder joint feature detection model and the solder joint state classification model, a comprehensive loss function is obtained, and the solder joint feature detection model and the solder joint state classification model are combined into a solder joint degradation classification model. The solder joint features in the external image set and the solder joint degradation features in the solder joint degradation state set are standardized feature processing to obtain solder joint feature vector and solder joint degradation feature vector. The solder joint feature vector and the solder joint degradation feature vector are sequentially subjected to feature alignment and element-by-element fusion operations to obtain an initial fused feature vector. A statistical learning method is used to select a fusion feature vector from the initial fusion feature vector; The integrated loss function is optimized using the fused feature vectors until the solder joint degradation classification model converges. Acquire the target solder joint image of the solder joint to be tested in the device under test, input the target solder joint image into the trained solder joint degradation classification model, and obtain the solder joint state classification result output by the solder joint degradation classification model; The solder joint degradation state of the solder joint to be tested is determined based on the solder joint status classification results.

2. The method for analyzing solder joint degradation state based on deep learning according to claim 1, characterized in that, The step of processing the solder joint positions of all the sample devices based on the solder joint position information and using the solder joint slicing method to acquire internal images of the solder joints of all the sample devices includes the following steps: Based on the solder joint location information, the solder joint locations of various sample devices are cut into solder joint slices according to preset dimensions; The weld joint slices are embedded into a preset standard template to form a weld joint standard template; The standard template for the weld joint is ground and polished. The image of the standard template of the solder joint after grinding and polishing is acquired by the solder joint image acquisition device as the internal image of the solder joint.

3. The method for analyzing solder joint degradation state based on deep learning according to claim 1, characterized in that, The steps of acquiring the target solder joint image in the device under test, inputting the target solder joint image into the trained solder joint degradation classification model, and obtaining the solder joint state classification result output by the solder joint degradation classification model include the following: Acquire the target solder joint image in the device under test; Preprocess the target solder joint image; The preprocessed target solder joint image is input into the trained solder joint feature detection model to obtain the target solder joint features output by the solder joint feature detection model; The target solder joint features are input into the trained solder joint state classification model to obtain the solder joint state classification result output by the solder joint state classification model.

4. A deep learning-based solder joint degradation state analysis system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 3.

5. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images