Solder joint defect detection methods, devices, electronic equipment and readable storage media

Abstract

This application provides a method, apparatus, electronic device, and readable storage medium for detecting weld joint defects. The method includes: processing a weld unit image to be processed according to a preset intersection detection algorithm to obtain a processed target weld unit image; inputting the target weld unit image into a first target model to determine global target features of the target weld unit image; and inputting the target weld unit image into a second target model to determine local target features of the target weld unit image; fusing the global and local target features according to a third target model to obtain target feature values, whereby the target feature values ​​represent the degree of loss in the target weld unit image; and performing weld joint defect detection based on the target feature values ​​to obtain the weld joint detection result of the weld unit. The solution of this application utilizes the fusion of global and local features to achieve weld joint defect detection, thereby improving the detection accuracy.

Description

Technical Field

[0001] This application relates to the field of solder joint inspection technology, and in particular to a solder joint defect inspection method, apparatus, electronic device and readable storage medium. Background Technology

[0002] With the development of electronic information, the scale of electronic product manufacturing has been increasing year by year, and the form of circuit boards is gradually developing towards more precise flexible circuits. Three-dimensional circuits are fabricated using laser wiring technology, which can realize more functions according to design requirements. They integrate the electrical interconnection, support, and protection functions of ordinary circuit boards onto a single device, resulting in smaller individual resistors and more precise designs. Currently, they are widely used in the automotive, industrial, computer, and communication fields. However, the assembly and soldering of three-dimensional circuits is difficult. A single circuit board contains a large number of micro-circuit components, involves many processing steps, and has poor quality consistency. After soldering, inspection is required, and defective products must be reworked to avoid greater economic losses. Commonly used methods for inspecting soldering quality include visual inspection, infrared thermal imaging inspection, automatic optical inspection, and electrical function testing.

[0003] Manual visual inspection of solder joints is often limited by human resources. The small size of solder joints in three-dimensional circuits necessitates the use of magnifying glasses and other equipment. Due to the miniaturization and increased density of electronic components, the inspection speed cannot keep pace with production speed, and it also negatively impacts the health of inspection personnel. Solder joint defects are diverse, and different types of defects share certain similarities, making manual inspection highly dependent on worker experience and subjectivity. Radiographic inspection estimates internal defects based on the attenuation of incident rays at the solder joint. However, using radiographic inspection for solder joint defects has a high failure rate, consumes significant resources, is time-consuming, and makes rework difficult.

[0004] Infrared thermal imaging analyzes potential defects by scanning and observing the surface temperature changes of the weld joint being tested. Infrared thermal imaging has high detection accuracy and reliability, but the detection time is long and the cost of the instrument is high, making it difficult to promote.

[0005] Automatic optical inspection, which is based on optical systems and machine vision inspection technology, acquires images of the object under test, compares the acquired images with template images, analyzes and processes them, and thus judges the defects of the object under test. However, it is cumbersome to operate and users need professional training to set up the processing algorithms accurately.

[0006] Electrical function testing involves energizing the device under test (DUT) under load conditions and then measuring data such as current and voltage in its circuit. However, electrical function testing is difficult to detect cold solder joints, and the process involves contact with the DUT, which can easily cause unnecessary damage.

[0007] In conclusion, under the modern production model that demands high efficiency and precision, existing methods for detecting weld defects can no longer meet production needs. Summary of the Invention

[0008] The purpose of this application is to provide a method, apparatus, electronic device and readable storage medium for detecting solder joint defects, so as to solve the drawbacks of existing methods such as manual visual inspection, infrared thermal imaging inspection, automatic optical inspection and electrical function inspection.

[0009] To achieve the above objectives, embodiments of this application provide a method for detecting weld joint defects, comprising:

[0010] Based on the preset intersection detection algorithm, the welding unit image to be processed is processed to obtain the processed target welding unit image;

[0011] The target welding unit image is input into a first target model to determine the global target features of the target welding unit image, and the target welding unit image is input into a second target model to determine the local target features of the target welding unit image;

[0012] According to the third target model, the global target features and the local target features are fused to obtain target feature values, which are used to represent the degree of loss of the target welding unit image;

[0013] Weld joint defects are detected based on the target feature values ​​to obtain the weld joint detection results of the welding unit.

[0014] Optionally, the welding unit image to be processed is processed according to a preset intersection detection algorithm to obtain a processed target welding unit image, including:

[0015] The image of the welding unit to be processed is preprocessed to obtain a first target image; the image preprocessing includes at least one of image grayscale processing, Gaussian filtering processing and morphological filtering processing.

[0016] The center point of the first target image is determined based on the corner points of the first target image.

[0017] Based on the intersection detection algorithm and the center point position, a preset pattern is drawn at the center point position to segment the first target image, thereby obtaining the target welding unit image.

[0018] Optionally, the target welding unit image is input into a first target model to determine the global target features of the target welding unit image, including:

[0019] The target welding unit image is input into the global feature extraction module of the first target model to determine the global features of the target welding unit image; the global feature extraction module includes multiple convolutional layers and one fully connected layer, and each convolutional layer has the same structure;

[0020] The global features are input into the first distance metric module of the first target model to determine the global target features of the target welding unit image;

[0021] The global target feature is used to represent at least one of the feature data of color feature, texture feature and shape feature of the target welding unit image; the first distance measurement module includes at least a preset first cosine similarity function.

[0022] Optionally, the target welding unit image is input into a second target model to determine the local target features of the target welding unit image, including:

[0023] The target welding unit image is input into the local feature extraction module of the second target model to determine the local features of the target welding unit image; the local feature extraction module includes multiple convolutional layers;

[0024] The local features of the target welding unit image are input into the second distance metric module of the second target model to determine the local target features of the target welding unit image;

[0025] The local target features are used to represent at least one of the edge features, corner features, and curve features of the target welding unit image; the second distance measurement module includes at least a preset second cosine similarity function.

[0026] Optionally, based on the third target model, the global target features and the local target features are fused to obtain target feature values, including:

[0027] Based on the global target features and the preset cross-entropy cost function, determine the first loss value of the global target features and the first loss weight corresponding to the first loss value;

[0028] Based on the local target features and the cross-entropy cost function, determine the second loss value of the local target features and the second loss weight corresponding to the second loss value;

[0029] The target feature value is determined by adding the product of the first loss value and the first loss weight, and the product of the second loss value and the second loss weight.

[0030] Optionally, before processing the welding unit image to be processed, the method further includes:

[0031] The sample data is enhanced using a preset method on the welding unit images to be processed;

[0032] The preset method includes at least one of the following:

[0033] A geometric transformation is performed on the welding unit image to be processed, the geometric transformation including at least one of flipping, shifting, deforming, scaling, and cropping;

[0034] The image of the welding unit to be processed is subjected to color transformation, which includes increasing or decreasing some color components of the image of the welding unit to be processed.

[0035] Optionally, weld joint defect detection is performed based on the target feature value to obtain the weld joint detection result of the welding unit, including:

[0036] The target loss value is input into a preset solder joint detection model for training to obtain the target solder joint detection model;

[0037] The image of the welding unit to be processed is input into the target solder joint detection model to determine whether there is a detection result of solder joint defects in the image of the welding unit to be processed; the solder joint defects include at least one of missing solder, excessive tin, and tombstoning.

[0038] To achieve the above objectives, embodiments of this application also provide a weld joint defect detection device, comprising:

[0039] The first processing module is used to process the welding unit image to be processed according to the preset intersection detection algorithm to obtain the processed target welding unit image.

[0040] The first determining module is used to input the target welding unit image into a first target model to determine the global target features of the target welding unit image, and to input the target welding unit image into a second target model to determine the local target features of the target welding unit image;

[0041] The second processing module is used to fuse the global target features and the local target features according to the third target model to obtain target feature values; the target feature values ​​are used to represent the degree of loss of the target welding unit image;

[0042] The third processing module is used to detect weld defects based on the target feature values ​​and obtain the weld detection results of the welding unit.

[0043] To achieve the above objectives, embodiments of this application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the solder joint defect detection method as described in any of the preceding claims.

[0044] To achieve the above objectives, embodiments of this application also provide a readable storage medium having a program or instructions stored thereon, characterized in that the program or instructions, when executed by a processor, implement the steps in the solder joint defect detection method as described in any of the preceding claims.

[0045] The beneficial effects of the above technical solution in this application are as follows:

[0046] In the embodiments of this application, the welding unit image to be processed is processed according to a preset intersection detection algorithm to obtain the processed target welding unit image. Here, the intersection detection algorithm is used to solve the problems of reflection, blurred edges, and high noise in the original welding point image, as well as the inability to use edge detection and thresholding methods for segmentation. The target welding unit image is input into a first target model to determine the global target features of the target welding unit image, and the target welding unit image is input into a second target model to determine the local target features of the target welding unit image. According to a third target model, the global target features and local target features are fused to obtain target feature values, which are used to represent the degree of loss of the target welding unit image. Welding point defect detection is performed based on the target feature values ​​to obtain the welding point detection result of the welding unit. This application utilizes the fusion of global features and local representations of welding points to improve the accuracy of welding point defect detection and provides a solution for automated welding point defect detection. Attached Figure Description

[0047] Figure 1 A schematic flowchart illustrating the solder joint defect detection method provided in this application embodiment;

[0048] Figure 2 A schematic diagram of the network structure of the global feature extraction module provided in the embodiments of this application;

[0049] Figure 3 A schematic diagram of the network structure of the local feature extraction module provided in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of the weld joint defect detection device provided in the embodiments of this application. Detailed Implementation

[0051] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0052] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0053] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0054] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0055] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0056] like Figure 1 As shown in the figure, this application provides a method for detecting weld joint defects, including:

[0057] Step 11: Process the welding unit image to be processed according to the preset intersection detection algorithm to obtain the processed target welding unit image.

[0058] It should be noted that the soldering unit mainly consists of three parts: solder pads, resistors, and solder resist. Solder joint defects include missing solder, excessive solder, tombstoning, etc. Solder joint defect detection mainly involves qualitative judgment of the soldering unit, that is, judging whether there are abnormalities in the solder pads, resistors, and solder resist to be inspected. It is unrelated to the background information in the circuit board. Therefore, it is necessary to segment the soldering unit from the circuit board image.

[0059] In this embodiment, the welding unit image to be processed is an image captured by a telecentric camera. This image needs to be processed before subsequent analysis can proceed. Common image segmentation methods include threshold-based segmentation and edge-based segmentation. However, due to issues such as reflection, blurred edges, and high noise in the original weld point image, edge detection and thresholding methods cannot be used for segmentation. To ensure accurate extraction of the welding unit image and the quality of subsequent weld point defect detection images, this application performs an overall analysis of the circuit board. The welding unit background exhibits relatively clear geometric relationships. A preset intersection detection algorithm is used to process the welding unit image, resulting in a processed target welding unit image with clearly defined and easily extractable features. This achieves welding unit image segmentation. The processed image is then labeled to complete the dataset creation, laying the foundation for subsequent network training.

[0060] Step 12: Input the target welding unit image into the first target model to determine the global target features of the target welding unit image; and input the target welding unit image into the second target model to determine the local target features of the target welding unit image.

[0061] Step 13: Based on the third target model, fuse the global target features and the local target features to obtain target feature values, which are used to represent the degree of loss of the target welding unit image.

[0062] In this embodiment, the global target features and local target features of the image have significant differences. The global target features can better summarize the higher-level semantic information of the image, while the local target features are more specific and have a stronger ability to describe the details of the image. According to the third target model, fusing the two can improve the accuracy of defect detection.

[0063] Step 14: Perform weld joint defect detection based on the target feature value to obtain the weld joint detection result of the welding unit.

[0064] In this embodiment, the target feature value is used to determine the final target solder joint detection model, and then the image of the welding machine unit is processed by the target solder joint detection model to obtain the solder joint detection result of the welding unit, thereby realizing the automated detection of solder joints in three-dimensional circuits.

[0065] Optionally, step 11 above includes:

[0066] The image of the welding unit to be processed is preprocessed to obtain a first target image; the image preprocessing includes at least one of image grayscale processing, Gaussian filtering processing and morphological filtering processing.

[0067] The center point of the first target image is determined based on the corner points of the first target image.

[0068] Based on the intersection detection algorithm and the center point position, a preset pattern is drawn at the center point position to segment the first target image, thereby obtaining the target welding unit image.

[0069] In this embodiment, to simplify the computational load of subsequent processes, the original image is first converted to grayscale. The grayscale image still contains a significant amount of noise, negatively impacting edge detection. To address this, a Gaussian function is convolved with a target function to smooth the image and remove noise. However, after noise removal, the image still contains numerous discontinuous regions. Erosion and dilation are two fundamental morphological operations that can isolate individual elements and connect different elements in an image. Before intersection detection, morphological opening and closing operations need to be performed on the image to obtain a filtered image. In other words, before intersection detection, the welding unit image to be processed undergoes grayscale conversion, Gaussian filtering for smoothing and denoising, and morphological filtering to remove noise, thereby obtaining the first target image.

[0070] After obtaining the first target image, four straight lines are drawn along the edge of the first target image to determine the corner point positions. The center point position of the first target image is determined by connecting the four corner points, which is the center point position of the welding unit. The center point position of the first target image is the intersection of the four straight lines passing through the corner points.

[0071] Based on the intersection detection algorithm and the center point position, a preset pattern is drawn at the center point position to segment the first target image. Preferably, a central rectangle is drawn at the center point position to segment the first target image, resulting in a target welding unit image. The target welding unit image determined by the intersection detection algorithm in this application provides technical support for subsequent acquisition of global and local features.

[0072] Optionally, in step 12 above, inputting the target welding unit image into the first target model and determining the global target features of the target welding unit image includes:

[0073] Step 121: Input the target welding unit image into the global feature extraction module of the first target model to determine the global features of the target welding unit image; the global feature extraction module includes multiple convolutional layers and one fully connected layer, and each convolutional layer has the same structure;

[0074] Step 122: Input the global features into the first distance metric module of the first target model to determine the global target features of the target welding unit image;

[0075] The global target feature is used to represent at least one of the feature data of color feature, texture feature and shape feature of the target welding unit image; the first distance measurement module includes at least a preset first cosine similarity function.

[0076] It should be noted that the global feature extraction module α of the first target model g It mainly consists of convolutional networks. The accuracy of the embedding representation of the input solder joint samples in the feature space directly affects the detection accuracy. Since the number of solder joint defect samples is small, an overly deep network will cause overfitting. In the process of network structure design, it is necessary to fully consider the accuracy of solder joint feature extraction and the setting of parameters.

[0077] It should also be noted that the above-mentioned determination of global target features uses a metric method to learn the distance between the feature vectors of each solder joint sample. Theoretically, solder joints of the same type have high similarity, while solder joints of different types have low similarity. The algorithm is designed to calculate the distance between points of the same type less than the distance between points of different types. The overall structure of the first target model consists of the global feature extraction module α. g and the first distance metric module β g It consists of two parts.

[0078] In this embodiment, the global feature extraction module α g It consists of four convolutional layers and one fully connected layer, as shown in the reference. Figure 2 As shown, convolutional layers C1 to C4 have the same structure. Each convolutional layer contains a normalization layer, an activation function layer, and a max pooling layer. Finally, a fully connected layer connects all neurons through weights, integrating the weld point image information extracted by the convolutional and pooling layers, and mapping the learned weld point feature representation to the label space of the weld point samples. In step 121 above, the target welding unit image is input into the global feature extraction module α of the first target model. g In this process, global features of the target welding unit image can be obtained.

[0079] In this embodiment of the application, the first distance measurement module β g Global features are extracted from solder joint images of the same category to construct a global feature description space. Within this space, the similarity between the solder joint image to be detected and each category is calculated, thereby enabling the detection of the solder joint. Considering the practicalities of solder joint defect detection, this application calculates the distance between similarity scores of solder joint samples using cosine similarity.

[0080] Here, the cosine similarity calculation uses the first cosine similarity function. Cosine similarity calculates the cosine of the angle between target sample vectors, ranging from -1 to 1. The closer the result is to 1, the higher the similarity between the two target objects. The formula for calculating the distance between vector A and vector B, i.e., the first cosine similarity function, is: That is, the product of vector A and vector B is divided by the product of the magnitudes of vector A and vector B; here, in step 122, the global features are input into the first distance metric module β. g In this method, the global features of the input object are array features, which can be used to obtain the global target features of the target welding unit image.

[0081] Furthermore, the global target features determined above can be used to determine the loss value of the global features.

[0082] Optionally, in step 12 above, inputting the target welding unit image into the second target model and determining the local target features of the target welding unit image includes:

[0083] Step 123: Input the target welding unit image into the local feature extraction module of the second target model to determine the local features of the target welding unit image; the local feature extraction module includes multiple convolutional layers;

[0084] Step 124: Input the local features of the target welding unit image into the second distance metric module of the second target model to determine the local target features of the target welding unit image;

[0085] The local target features are used to represent at least one of the edge features, corner features, and curve features of the target welding unit image; the second distance measurement module includes at least a preset second cosine similarity function.

[0086] It should be noted that the structure of the solder joint local representation learner is inspired by the deep nearest neighbor neural network algorithm based on metric learning. Since each type of defect in a solder joint has only a very small number of samples, using pooling features and fully connected layer features designed for large-scale images to solve the defect detection problem has drawbacks. Encoding or quantizing the local representation of a solder joint image to obtain a concise image feature will result in the loss of much discriminative information beneficial for defect detection, and this loss is irreversible.

[0087] The local representation of each defect category should be fully utilized to more effectively express the defect category of the solder joint. To this end, this application designs a second objective model for utilizing the local representation of solder joint defects. Its core difference from the network structure mentioned above lies in that the classification layer of the deep network is based on local representation for measurement. The overall model structure is similar to traditional metric learning, consisting of a feature extraction module and a distance measurement module.

[0088] In this embodiment of the application, the local feature extraction module α of the second target model l Similar to the global feature-based solder joint detection algorithm, the local feature extraction module α... l It is also composed of convolutional neural networks, but unlike convolutional neural networks, it does not contain fully connected layers. Theoretically, any suitable convolutional neural network can be used for this module. This application primarily considers the practical needs of solder joint defect detection in its network structure design, focusing on the local feature extraction module α of the second target model. l Network architecture reference Figure 3 As shown, a four-layer convolutional neural network is used as the feature extraction module. It mainly consists of four convolutional modules, each containing a convolutional layer, a batch normalization layer, and an activation function layer. In addition, a 2×2 max pooling layer is added after the first two convolutional modules.

[0089] For example, the target welding unit image X, and the local feature extraction module α l It will output a three-dimensional tensor h×w×d, where h represents the height of the image, w represents the width of the image, and d represents the depth of the image. It can be viewed as consisting of h×w local descriptors with feature dimension d, in the following form:

[0090]

[0091] Where x i Let m represent the i-th local descriptor, where m = h × w.

[0092] In step 123 above, the target welding unit image is input into the local feature extraction module α. l This allows us to directly obtain the local features of the target welding unit image.

[0093] Furthermore, the second distance metric module β l Local representations of solder joint images of the same category are extracted to construct a local representation description space. In this space, the similarity between the solder joint image to be detected and each category is calculated, and then the solder joint to be detected is detected.

[0094] Specifically, taking a solder joint 'a' to be tested as an example, the local feature extraction module α... lLet a be represented as a set of local characterization descriptors. For each descriptor x of a i Find the corresponding descriptors in each of the pre-learned categories. Calculate x i and The similarity between the categories is calculated by summing the results to obtain the similarity measure between 'a' and each category. The specific formula for the second cosine similarity function can be expressed as follows:

[0095]

[0096] Here, cos(·) represents the cosine similarity metric function.

[0097] It should be noted that the above input is given to the local feature extraction module α. l The object is a vector, meaning the local features of the target welding unit image are vectors. These local features are then input into the second distance metric module β. l This allows us to determine the local target features of the target welding unit image.

[0098] Optionally, step 13 above includes:

[0099] Based on the global target features and the preset cross-entropy cost function, determine the first loss value of the global target features and the first loss weight corresponding to the first loss value;

[0100] Based on the local target features and the cross-entropy cost function, determine the second loss value of the local target features and the second loss weight corresponding to the second loss value;

[0101] The target feature value is determined by adding the product of the first loss value and the first loss weight, and the product of the second loss value and the second loss weight.

[0102] In this embodiment, based on the global target features and the preset cross-entropy cost function, the first loss value L of the global target features can be obtained. g By combining the local target features and the cross-entropy cost function, the second loss value L1 for the local target features can be determined. Since the global target features and local target features can be mapped onto a chessboard, the first loss weight A corresponding to the first loss value can be determined by searching the grid on the chessboard. g And, determine the second loss weight A1 corresponding to the second loss value. Furthermore, determine the target feature value L according to the following formula. A Target eigenvalue L A This represents the final loss value.

[0103] That is, LA =A g L g +A l L l .

[0104] Where A represents the loss weight of the feature learner, which is between 0 and 1 (inclusive), and A g and A l The sum of these values ​​equals 1. The loss weights define the contribution of L to the total loss function. l and L g These respectively represent the importance of local representations or global features; L l and L g These represent the loss values ​​for detection using local and global features, respectively.

[0105] It should be noted that in A g When A1 is 0.15 and 0.85 respectively, the target eigenvalue L A The trained weld defect detection model achieved optimal performance.

[0106] Furthermore, the aforementioned pre-defined cross-entropy cost function can be expressed as:

[0107]

[0108] Among them, y i This is the label value for the solder joint (1 for a correct detection, 0 for an incorrect detection). Let n represent the probability of a correct solder joint detection, and n represent the total number of training data.

[0109] Optionally, before step 11 above, the following steps are also included:

[0110] The sample data is enhanced using a preset method on the welding unit images to be processed;

[0111] The preset method includes at least one of the following:

[0112] A geometric transformation is performed on the welding unit image to be processed, the geometric transformation including at least one of flipping, shifting, deforming, scaling, and cropping;

[0113] The image of the welding unit to be processed is subjected to color transformation, which includes increasing or decreasing some color components of the image of the welding unit to be processed.

[0114] In this embodiment of the application, due to the high cost of weld point samples and the difficulty in obtaining defect detection sample data, it is difficult to have sufficient data to complete the learning. While seeking more data, the sample data is enhanced using a preset method based on the welding unit image to be processed.

[0115] Specific preset methods include at least one of the following:

[0116] Single-sample data augmentation is used to expand solder joint samples. The expansion only operates on the sample itself, primarily involving geometric transformations and color transformations. Geometric transformations of the solder joint samples mainly include operations such as flipping, shifting, deforming, scaling, and cropping. Color transformations mainly involve adding or removing certain color components of the solder joint.

[0117] After data augmentation, the dataset can contain thousands of data samples of 12 types of defects, including tombstoning, missing solder (including both through-hole and surface mount resistors), missing resistors, resistor flipping, excessive solder, and resistor misalignment, as well as qualified surface mount and through-hole resistors.

[0118] Optionally, step 14 above includes:

[0119] The target loss value is input into a preset solder joint detection model for training to obtain the target solder joint detection model;

[0120] The image of the welding unit to be processed is input into the target solder joint detection model to determine whether there is a detection result of solder joint defects in the image of the welding unit to be processed; the solder joint defects include at least one of missing solder, excessive tin, and tombstoning.

[0121] In this embodiment, a target weld point detection model is trained based on a predetermined target loss value. The target weld point detection model is then obtained, and the detection result of whether the welding unit image to be processed has weld point defects can be determined through the final trained target weld point detection model, thereby improving the accuracy of welding unit image detection.

[0122] In one specific implementation, the hardware environment of the detection platform of this application consists of an image acquisition device (object-side telecentric lens, industrial camera), a mechanical support structure, a translation stage, and a computer. The image acquisition device mainly consists of an industrial camera, an object-side telecentric lens, and a coaxial light source.

[0123] A Microview SW-A5005-PGC24 color camera was selected, with a Canrui Optics XF-10MDT2X65D-1C-VI object-side telecentric lens. Specific parameters are shown in Tables 1 and 2 below. Due to severe reflections from the circuit under test, a light-diffusing filter was added to ensure image quality and reduce noise. The coaxial light source used was a Light Chaser FL-PI4406-10. Because the addition of the light-diffusing filter reduced light transmittance, a 10W coaxial white light was selected to ensure bright and uniform illumination across the entire field of view.

[0124] Table 1 Main parameters of industrial cameras

[0125]

[0126] Table 2 Main parameters of the telecentric lens

[0127]

[0128] In metric learning, the dataset partitioning and training methods differ from those in conventional deep learning. Conventional deep learning datasets are divided into training and testing sets, while metric learning datasets are divided into several smaller sets of conventional datasets, each containing K samples from N classes. Model training is performed on each set of data to generalize even with limited samples. This application addresses the limitation of solder joint image data by partitioning the dataset into S sets at a 10:1 ratio. tr and S t Two parts, S tr Used for training on solder joint defects, S t Used for testing weld joint defects. The model is trained in groups, with each group starting from S... tr Ten images are extracted from each image category in S. t Extract 10 images of a certain type from the dataset. After each group is calculated, return the average accuracy and loss value.

[0129] The network is programmed using a specific machine environment, such as a Python language environment, on an Ubuntu 20.0 environment. The algorithm is implemented based on the PyTorch framework. Parameters are set as follows: epoch set to 50, learning rate set to 0.01, and learning decay coefficient set to 1×10⁻⁶. -6 The batch size is set to 64, the ADMA optimizer is used, and the weight coefficient A is... g A l The adjustment interval is set to 0.05, and a grid search method is used for optimization. Adam is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the neural network weights based on the training data.

[0130] In A g A l When the values ​​are 0.15 and 0.85 respectively, the weld joint defect detection model achieves optimal performance. As the number of training iterations increases, the loss value decreases continuously. When the number of training iterations reaches about 40, the accuracy and loss value basically no longer change, indicating that the model training is complete. The accuracy of weld joint defect detection is 98.4%, and the loss value is 0.212.

[0131] The method in this application is compared with traditional Support Vector Machines (SVM), conventional Convolutional Neural Networks (CNNs), and two baseline network algorithms (e.g., Matching Network and DN4, where DN4 is a metric-based few-shot algorithm). The results are shown in Table 3. Due to the small sample size and limited features, the Gaussian kernel function is chosen in the SVM algorithm. The low accuracy in solder joint defect detection is because SVM cannot comprehensively capture the features of each class of samples, and noise in the solder joint defect images significantly impacts the model. In Convolutional Neural Networks (CNNs), the "VGG-16" model with smaller convolutional and pooling kernels and more convolutional layers is selected. While its structure is simple and its generalization performance is good, the neural network's recognition performance on this dataset is low, mainly because the number of samples in the solder joint defect dataset is small, making the network prone to overfitting. Metric learning and other few-shot learning training strategies are better suited to the application scenario of this application. The Matching Network has a good grasp of global features, but it is difficult to distinguish between different defect types with high similarity in solder joints.

[0132] Deep nearest neighbor networks have good control over details. The algorithm designed in this application integrates global features and local representations of the solder joint, which greatly improves the accuracy of defect detection.

[0133] Table 3 Comparison of results between the method of this application and other methods

[0134]

[0135] In summary, based on the need for automated detection of circuit solder joint defects, this application proposes a segmentation algorithm for solder unit images to accurately extract solder joint regions, laying the foundation for subsequent defect detection. Combining metric learning knowledge, a solder joint defect detection algorithm is designed, and the accuracy of the method is verified through experiments. The proposed method solves the problems of difficult segmentation of solder unit images and the large variety and small differences in solder joint defects, realizing automated detection of three-dimensional circuit solder joints. However, due to issues such as imbalanced solder joint samples, the accuracy of network recognition is difficult to improve. Further research can be conducted by expanding the solder joint sample and designing and manufacturing mature automated solder joint defect detection equipment.

[0136] Reference Figure 4 As shown in the figure, this application embodiment also provides a weld joint defect detection device, including:

[0137] The first processing module 41 is used to process the welding unit image to be processed according to the preset intersection detection algorithm to obtain the processed target welding unit image.

[0138] The first determining module 42 is used to input the target welding unit image into a first target model to determine the global target features of the target welding unit image, and to input the target welding unit image into a second target model to determine the local target features of the target welding unit image;

[0139] The second processing module 43 is used to fuse the global target features and the local target features according to the third target model to obtain target feature values; the target feature values ​​are used to represent the degree of loss of the target welding unit image;

[0140] The third processing module 44 is used to perform weld joint defect detection based on the target feature value to obtain the weld joint detection result of the welding unit.

[0141] Optionally, the first processing module 41 described above includes:

[0142] The first processing unit is used to perform image preprocessing on the welding unit image to be processed to obtain a first target image; the image preprocessing includes at least one of image grayscale processing, Gaussian filtering processing and morphological filtering processing.

[0143] The first determining unit is used to determine the center point position of the first target image based on the corner point positions of the first target image;

[0144] The second processing unit is used to segment the first target image by drawing a preset pattern at the center point position according to the intersection detection algorithm and the center point position, so as to obtain the target welding unit image.

[0145] Optionally, the first determining module 42 described above includes:

[0146] The second determining unit is used to input the target welding unit image into the global feature extraction module of the first target model to determine the global features of the target welding unit image; the global feature extraction module includes multiple convolutional layers and one fully connected layer, and each convolutional layer has the same structure;

[0147] The third determining unit is used to input the global features into the first distance measurement module of the first target model to determine the global target features of the target welding unit image;

[0148] The global target feature is used to represent at least one of the feature data of color feature, texture feature and shape feature of the target welding unit image; the first distance measurement module includes at least a preset first cosine similarity function.

[0149] Optionally, the first determining module 42 described above further includes:

[0150] The fourth determining unit is used to input the target welding unit image into the local feature extraction module of the second target model to determine the local features of the target welding unit image; the local feature extraction module includes multiple convolutional layers;

[0151] The fifth determining unit is used to input the local features of the target welding unit image into the second distance measurement module of the second target model to determine the local target features of the target welding unit image;

[0152] The local target features are used to represent at least one of the edge features, corner features, and curve features of the target welding unit image; the second distance measurement module includes at least a preset second cosine similarity function.

[0153] Optionally, the second processing module 43 described above includes:

[0154] The sixth determining unit is used to determine the first loss value of the global target feature and the first loss weight corresponding to the first loss value based on the global target feature and the preset cross-entropy cost function.

[0155] The seventh determining unit is used to determine, based on the local target features and the cross-entropy cost function, a second loss value of the local target features and a second loss weight corresponding to the second loss value;

[0156] The eighth determining unit is used to add the product of the first loss value and the first loss weight, and the product of the second loss value and the second loss weight, to determine the target feature value.

[0157] Optionally, the above-mentioned weld joint defect detection device further includes:

[0158] The fourth processing module is used to enhance the sample data of the welding unit image to be processed using a preset method;

[0159] The preset method includes at least one of the following:

[0160] A geometric transformation is performed on the welding unit image to be processed, the geometric transformation including at least one of flipping, shifting, deforming, scaling, and cropping;

[0161] The image of the welding unit to be processed is subjected to color transformation, which includes increasing or decreasing some color components of the image of the welding unit to be processed.

[0162] Optionally, the third processing module 44 described above includes:

[0163] The third processing unit is used to input the target loss value into a preset solder joint detection model for training, thereby obtaining the target solder joint detection model.

[0164] The fourth processing unit is used to input the welding unit image to be processed into the target solder joint detection model and determine whether the welding unit image to be processed has a detection result of solder joint defects; the solder joint defects include at least one of missing solder, excessive solder, and tombstoning.

[0165] The implementation embodiments of the above-mentioned weld joint defect detection method are all applicable to the embodiments of the weld joint defect detection device, and can achieve the same technical effect.

[0166] This application also provides an electronic device, characterized in that it includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the solder joint defect detection method as described in any of the above claims and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0167] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the solder joint defect detection method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0168] The processor mentioned above is the processor used in the solder joint defect detection method described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0170] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0171] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0172] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0173] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting weld joint defects, characterized in that, include: The sample data is enhanced using a preset method on the welding unit images to be processed; The preset method includes at least one of the following: performing geometric transformation on the welding unit image to be processed, the geometric transformation including at least one of flipping, shifting, deforming, scaling, and cropping; performing color transformation on the welding unit image to be processed, the color transformation including increasing or decreasing some color components of the welding unit image to be processed; According to a preset intersection detection algorithm, the welding unit image of the three-dimensional circuit to be processed is processed to obtain the processed target welding unit image, including: performing image preprocessing on the welding unit image to be processed to obtain a first target image; the image preprocessing includes at least one of image grayscale processing, Gaussian filtering processing, and morphological filtering processing; after obtaining the first target image, four straight lines are drawn along the edge of the first target image to determine the corner point positions, and the center point position of the first target image is determined by connecting the four corner points; the center point position of the first target image is the intersection of the four straight lines passing through the corner points; according to the intersection detection algorithm and the center point position, a preset pattern is drawn at the center point position to segment the first target image to obtain the target welding unit image; The target welding unit image is input into a first target model to determine the global target features of the target welding unit image. The target welding unit image is also input into a second target model to determine the local target features of the target welding unit image. The first target model includes a global feature extraction module and a first distance metric module. The global feature extraction module consists of four convolutional layers and one fully connected layer. Each convolutional layer includes a normalization layer, an activation function layer, and a max-pooling layer. The global target features represent at least one of the color, texture, and shape features of the target welding unit image. The data includes a first distance metric module that calculates feature similarity using a first cosine similarity function to output global target features; a second target model that includes a local feature extraction module and a second distance metric module. The local feature extraction module consists of four convolutional layers without fully connected layers. Each layer contains a convolutional layer, batch normalization, and an activation function. A max pooling layer is added after the first two layers. The local target features are used to represent at least one of the edge features, corner features, and curve features of the target welding unit image. The second distance metric module accumulates local descriptor similarity using a second cosine similarity function to output local target features. According to the third target model, the global target features and the local target features are fused to obtain target feature values, which are used to represent the degree of loss of the target welding unit image; Solder joint defect detection is performed based on the target feature value to obtain the solder joint detection result of the welding unit, including: inputting the target feature value into a preset solder joint detection model for training, and obtaining the target solder joint detection model by a small sample group training method of extracting 10 samples for each class; inputting the image to be detected into the target solder joint detection model, and outputting defect judgment results including missing solder, excessive solder, and tombstoning.

2. The method according to claim 1, characterized in that, The target welding unit image is input into the first target model to determine the global target features of the target welding unit image, including: The target welding unit image is input into the global feature extraction module of the first target model to determine the global features of the target welding unit image; The global features are input into the first distance metric module of the first target model to determine the global target features of the target welding unit image.

3. The method according to claim 1, characterized in that, The target welding unit image is input into the second target model to determine the local target features of the target welding unit image, including: The target welding unit image is input into the local feature extraction module of the second target model to determine the local features of the target welding unit image; The local features of the target welding unit image are input into the second distance metric module of the second target model to determine the local target features of the target welding unit image.

4. The method according to claim 1, characterized in that, Based on the third target model, the global target features and the local target features are fused to obtain target feature values, including: Based on the global target features and the preset cross-entropy cost function, determine the first loss value of the global target features and the first loss weight corresponding to the first loss value; Based on the local target features and the cross-entropy cost function, determine the second loss value of the local target features and the second loss weight corresponding to the second loss value; The target feature value is determined by adding the product of the first loss value and the first loss weight, and the product of the second loss value and the second loss weight.

5. A weld joint defect detection device, characterized in that, The apparatus, used in the weld joint defect detection method as described in any one of claims 1 to 4, comprises: The first processing module is used to process the welding unit image to be processed according to the preset intersection detection algorithm to obtain the processed target welding unit image. The first determining module is used to input the target welding unit image into a first target model to determine the global target features of the target welding unit image, and to input the target welding unit image into a second target model to determine the local target features of the target welding unit image; The second processing module is used to fuse the global target features and the local target features according to the third target model to obtain target feature values; the target feature values ​​are used to represent the degree of loss of the target welding unit image; The third processing module is used to detect weld defects based on the target feature values ​​and obtain the weld detection results of the welding unit.

6. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the solder joint defect detection method as described in any one of claims 1 to 4.

7. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the solder joint defect detection method as described in any one of claims 1 to 4.

Images