PCB welding spot defect detection method and device based on microscopic image

Abstract

The invention discloses a PCB welding spot defect detection method and device based on a microscopic image, and relates to the technical field of microscopic image recognition. The method comprises the following steps: acquiring a welding spot defect microscopic image by adopting a microscope, and carrying out preprocessing and labeling processing on the acquired welding spot defect microscopic image to generate a welding spot image data set; constructing a welding spot defect detection model based on the instance segmentation network model, wherein the welding spot defect detection model comprises a backbone network, a neck network and a detection head; in the backbone network, a C2FRF module is adopted to replace a C3k2 module; in the neck network, a P2-FPN structure is formed for feature fusion; and a joint loss function containing a bounding box regression loss function and a focus loss function based on a dynamic focusing mechanism is adopted to guide training of the welding spot defect detection model, and a welding spot defect detection result is obtained. The problems that in the prior art, when welding spot defects are detected, the detection accuracy is not high, a printed circuit board is interfered, and the detection cost is high are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of microscopic image recognition, and particularly relates to a PCB solder joint defect detection method and device based on a microscopic image. BACKGROUND

[0002] A printed circuit board (PCB) is a basic connecting component of electronic product assembly parts, is crucial in the integrated circuit industry, and is the foundation of the development of intelligent manufacturing and advanced manufacturing fields. With the development of integrated circuit technology, electronic devices and PCBs are becoming smaller in size, and the connection density and component integration are increasing, which poses great challenges to the reliability and yield control of PCB production and processing.

[0003] Traditional PCB bare board and solder joint detection technologies include manual detection, electrical detection, and infrared detection methods. Manual detection relies on human experience for judgment, electrical detection detects defects through electrical characteristic analysis, and infrared detection uses infrared technology to find abnormalities. However, manual detection is low in efficiency, is easily affected by subjective factors, and has limited detection capability for complex and small defects; electrical detection may interfere with the circuit board and cannot directly present the defect morphology; and infrared detection has poor detection effect on some non-temperature-related defects and has high equipment cost.

[0004] Therefore, how to perform solder joint defect detection to improve the accuracy and efficiency of detection and reduce the interference with and detection cost of the printed circuit board has become a technical problem to be solved. SUMMARY

[0005] In the embodiments of the present application, by providing a PCB solder joint defect detection method based on a microscopic image, the problem of low detection accuracy and efficiency, interference with the printed circuit board, and high detection cost in the prior art when performing solder joint defect detection is solved.

[0006] In the first aspect, the embodiment of the present application provides a PCB solder joint defect detection method based on microscopic images, the method comprising: using a microscope to obtain a solder joint defect microscopic image containing a global stereoscopic morphology, preprocessing and annotating the obtained solder joint defect microscopic image, and generating a solder joint image dataset; constructing a solder joint defect detection model based on an instance segmentation network model, the solder joint defect detection model comprising a backbone network, a neck network, and a detection head; using a C2FRF module instead of a C3k2 module in the backbone network to optimize the input feature map, and inputting the optimized feature map into the neck network; in the neck network, introducing the feature maps of levels 2 to 5 in the optimized feature map into a feature pyramid to form a detection head; A P2-FPN structure is formed for feature fusion to obtain a multi-level fusion feature map, and the multi-level fusion feature map is input into the detection head; a parameter sharing method and a depth-separable convolution lightweight detection head are adopted in the detection head to convert the multi-level fusion feature map into a solder joint defect detection result; a joint loss function including a bounding box regression loss function based on a dynamic focusing mechanism and a focus loss function is used to guide the training of the solder joint defect detection model; the solder joint image dataset is input into the solder joint defect detection model for training to obtain a model weight file; a microscope is used to take a microscopic image of the solder joint defect to be detected, and the model weight file is used to test the microscopic image of the solder joint defect to be detected to obtain the solder joint defect detection result.

[0007] In one possible implementation, the preprocessing and labeling of the acquired solder joint defect microscopic images to generate a solder joint image dataset includes: preprocessing the acquired solder joint defect microscopic images using a method combining contrast-limited adaptive histogram equalization and bilateral filtering; classifying the solder joint morphology in the solder joint defect microscopic images into four categories: normal solder quantity, excessive solder, insufficient solder, and solder tip, to obtain true labels; polygonal labeling of the solder joints, creating masks with different contours according to the category to which each solder joint belongs, to label the acquired solder joint defect microscopic images to obtain true frames; and data enhancement of the labeled solder joint defect microscopic images, and converting them into YOLO format to generate a solder joint image dataset.

[0008] In a possible implementation, the C2FRF module is used in the backbone network to replace the C3k2 module, the input feature map is optimized, and the optimized feature map is input to the neck network, including: using the FRF module to replace the original Bottleneck module to obtain the C2FRF module, the FRF module strengthens the information elements in the feature by introducing the PConv operation, and the specific operation process includes: using the PConv operation to operate the input feature map Perform calculations to reduce redundant channel operations; use the first linear projection of 1×1 convolution Adjust the dimension and enhance nonlinearity through GELU activation function to obtain intermediate features ; split the intermediate feature into a first feature and a second feature along the channel ; , the Reshape transformation is applied to the second feature , a depth separable convolution operation is applied to extract spatial features, and an attention map is generated through a Flatten operation, which is element-wise multiplied with the first feature to realize feature modulation and obtain an intermediate feature after feature modulation; the second linear projection of 1x1 convolution is performed on the intermediate feature after feature modulation to linearly transform the feature information of different dimensions, and the optimized feature map is output after activation by a GELU activation function .

[0009] In a possible implementation, in the neck network, the features of levels 2 to 5 in the optimized feature map are introduced into a feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fused feature map is obtained, including: using nearest neighbor interpolation upsampling to enlarge the size of the feature map of a low level to that of an adjacent high level Figure One ; using dynamic step size convolution downsampling to reduce the size of the feature map of a high level to that of an adjacent low level Figure One ; concatenating the feature maps of adjacent low levels and inputting them into a C2FRF module to fuse cross-stage features to obtain a multi-level fused feature map.

[0010] In a possible implementation, in the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to convert the multi-level fused feature map into a welding defect detection result, including: the parameter sharing method is designed to share the same detection head parameters for the multi-level fused feature map; a depth separable convolution and a CBS module are used to construct a lightweight detection head to convert the multi-level fused feature map into a welding defect detection result.

[0011] In a possible implementation, the joint loss function containing a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function is used to guide the training of the welding defect detection model, including: the expression of the bounding box regression loss function based on the dynamic focusing mechanism is: ; wherein, is the bounding box regression loss based on the dynamic focusing mechanism, is a non-monotonic focusing coefficient, , and is a hyperparameter for calculating the non-monotonic focusing coefficient, is the outlying degree of the anchor box, , is the gradient gain, is an exponential moving average value of is a bounding box regression loss function based on IoU, wherein IoU is an evaluation index for measuring the degree of overlap between a predicted box and a real box, is a penalty term; ; wherein, is an exponential calculation, is the horizontal coordinate of the predicted box, is the vertical coordinate of the predicted box, is the horizontal coordinate of the real box, is the vertical coordinate of the real box, is the width of the minimum closed box formed by the predicted box and the real box, is the height of the minimum closed box formed by the predicted box and the real box, is to separate the result of from the calculation graph; the expression of the focal loss function is: ; wherein, is the focal loss function, is the class balance coefficient, is the matching degree of the predicted probability and the real label, is the weight of the difficult sample; the expression of the focal loss function is: ; wherein, is the focal loss function, is the class balance coefficient, is the matching degree of the predicted probability and the real label, is the weight of the difficult sample; the expression of the joint loss function is: ; wherein, is the joint loss function, is a hyperparameter.

[0012] ​In a second aspect, the embodiments of the present application provide a PCB solder joint defect detection device based on microscopic images. The device comprises: a processing module configured to acquire a solder joint defect microscopic image containing a global three-dimensional morphology by using a microscope, to pre-process and label the acquired solder joint defect microscopic image, and to generate a solder joint image dataset; a construction module configured to construct a solder joint defect detection model based on an instance segmentation network model, the solder joint defect detection model comprising a backbone network, a neck network, and a detection head; in the backbone network, a C2FRF module is used to replace a C3k2 module to optimize the input feature map, and the optimized feature map is input into the neck network; in the neck network, the 2-5 level feature maps in the optimized feature map are introduced into a feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fused feature map is obtained, which is input into the detection head; in the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to convert the multi-level fused feature map into a solder joint defect detection result; a guiding module configured to guide the training of the solder joint defect detection model by using a joint loss function comprising a bounding box regression loss function based on a dynamic focus mechanism and a focal loss function; a training module configured to input the solder joint image dataset into the solder joint defect detection model for training to obtain a model weight file; a solder joint defect microscopic image to be detected is captured by using the microscope, and the model weight file is used to test the solder joint defect microscopic image to be detected to obtain a solder joint defect detection result.

[0013] In a third aspect, the embodiments of the present application provide a PCB solder joint defect detection server based on microscopic images, comprising a memory and a processor; the memory is configured to store computer executable instructions; the processor is configured to execute the computer executable instructions to implement the method of the first aspect or any possible implementation manner of the first aspect.

[0014] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores executable instructions, and a computer executes the executable instructions to implement the method of the first aspect or any possible implementation manner of the first aspect.

[0015] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects: The embodiment of the present application provides a PCB solder joint defect detection method based on a microscopic image. The method adopts a microscope to obtain a solder joint defect microscopic image containing a global three-dimensional morphology, provides rich and detailed original data for detection, can clearly present various characteristics of the solder joint, and lays a foundation for accurate detection. In the main network, a C2FRF module is used to replace a C3k2 module to optimize the input feature map, enhances the feature extraction capability, and enables the model to better capture the key characteristics of the solder joint defect. The neck network introduces a P2-FPN structure to perform feature fusion on the feature maps of 2 to 5 levels, realizes effective combination of different level features, enables the model to understand the solder joint features from multiple scales, and greatly improves the recognition accuracy of various solder joint defects. In the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to reduce the calculation amount and the parameter quantity of the model, accelerate the inference speed of the model, enable the model to quickly give a detection result when processing a large number of to-be-detected images, and significantly improve the detection efficiency. The joint loss function containing the boundary box regression loss function based on the dynamic focusing mechanism and the focal loss function is used to guide the model training, and the combination can more accurately measure the difference between the prediction result and the real label, promotes the model to learn more discriminative features, and further improves the detection accuracy. The present application detects based on the solder joint defect microscopic image, and belongs to a non-contact detection method. Compared with the traditional electrical detection method, the present application does not need to apply an electrical signal such as an electric current or a voltage to the circuit board, avoids potential damage to the circuit board caused by electrical interference, and guarantees the integrity and reliability of the circuit board. The automatic detection method based on deep learning reduces the dependence on artificial experience judgment and reduces the labor cost. Meanwhile, the automatic detection process can improve the consistency and stability of detection and reduce false detection and missed detection caused by human factors. The microscope is used to obtain the image, and compared with some special detection equipment with high cost, the microscope has the advantages of relatively low cost and relatively simple operation, can meet the detection requirements, and effectively reduces the investment cost of the detection equipment. The present application solves the problems of low detection accuracy and efficiency, interference to the printed circuit board and high detection cost in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0017] Figure 1 A flowchart of a PCB solder joint defect detection method based on a microscopic image provided by the embodiment of the present application; Figure 2 A schematic diagram of creating masks with different contours according to the category to which each solder joint belongs, provided in an embodiment of the present application; Figure 3 A schematic diagram of an improved detection head of the original YOLO11n-seg model provided in an embodiment of the present application; Figure 4 This is a precision-recall curve of the original YOLO11n-seg model provided in the embodiment of this application; Figure 5 A precision-recall curve of the solder joint defect detection model provided in the application embodiment; Figure 6 A schematic diagram showing visualization of solder joint defect detection results of a solder joint defect detection model provided in an embodiment of the present application; Figure 7 A schematic diagram of a PCB solder joint defect detection device based on microscopic images provided in an embodiment of the present application; Figure 8 A schematic diagram of a microscopic image-based PCB solder joint defect detection server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0020] The present application embodiment provides a PCB solder joint defect detection method based on microscopic images, such as Figure 1 As shown, the method includes steps S101 to S108. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a PCB solder joint defect detection method based on microscopic images. In the case that the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0021] S101: Obtain a micro image of a solder joint defect containing a global three-dimensional morphology using a microscope, and preprocess and label the obtained micro image of the solder joint defect to generate a solder joint image dataset.

[0022] Specifically, to obtain comprehensive and representative solder joint images, the present application collects a patch component soldering practice board, i.e., a printed circuit board (PCB) for soldering, and photographs it using a microscope. Considering that different patch components have different sizes, to improve spatial resolution and display more details, the magnification value and working distance of the microscope are dynamically adjusted, with the smallest component set to the highest magnification. Since it is difficult to accurately present the precise morphology of the solder joint in an overhead photograph, the patch component solder joint is photographed at a 45-degree angle through the shaft side of the microscope. This approach can better capture the spatial relationship between the solder joint surface and the background to obtain a micro image of a solder joint defect containing a global three-dimensional morphology, and make the three-dimensional morphology of the solder joint more intuitive in a two-dimensional image, providing more comprehensive feature information for the deep learning model of the present application, i.e., the solder joint defect detection model, and improving the recognition ability of complex defects. Ultimately, the present application obtains 450 micro images of solder joint defects under a micro perspective, which cover defective and non-defective solder joints, and the defect types include excessive soldering, insufficient soldering, and soldering tip pulling, etc.

[0023] The obtained micro image of the solder joint defect is preprocessed and labeled to generate a solder joint image dataset, including the following contents.

[0024] The obtained micro image of the solder joint defect is preprocessed using a method combining contrast-limited adaptive histogram equalization and bilateral filtering.

[0025] Specifically, during the tilting photographing process of the microscope, the reflection of the solder joint surface and the shadow effect can easily lead to uneven distribution of image contrast, and part of the solder joint features can be covered by overexposure of bright parts or loss of dark details. The present application introduces a method combining contrast-limited adaptive histogram equalization and bilateral filtering. Although traditional global histogram equalization can improve the overall contrast, it has the problem of over-enhancement, especially in the high-contrast region of the solder joint, which can introduce artifacts or noise, interfering with subsequent defect detection. The application of contrast-limited adaptive histogram equalization not only improves the light-dark contrast of the solder joint region in the image, but also significantly improves the visibility and detail expression of the solder joint defect; the application of bilateral filtering smoothes the noise in the image while better preserving the edges and details of the solder joint, providing a clearer and more reliable image input for subsequent defect detection and solder joint segmentation, and improving the detection accuracy and robustness of the model.

[0026] The solder joint morphology in the micro image of the solder joint defect is divided into four categories: normal soldering, excessive soldering, insufficient soldering, and soldering tip pulling, and real labels are obtained.

[0027] The welding points are polygonally labeled, and different contour masks are created according to the categories to which each welding point belongs, so as to label and process the obtained welding defect microscopic images and obtain the real frame.

[0028] Specifically, the labeling process is carried out according to the patch component welding quality inspection specification. The welding quality of the welding points in the data set is classified and labeled, and the shape of the welding points in the image is divided into four categories: normal soldering amount, excessive soldering amount, insufficient soldering amount and soldering tip. In this way, the real label is obtained. In the labeling method, the open source image annotation tool named LabelMe is used to polygonally label the obtained welding defect microscopic images, and the labeling method will generate a corresponding JSON format (a lightweight data exchange format) file for each instance object. The instance object in the present application is a single welding point in each welding defect microscopic image.

[0029] Figure 2 The schematic diagram of creating different contour masks according to the categories to which each welding point belongs is provided for the embodiments of the present application. Figure 2 (a) in the figure is a mask constructed for a welding point with a normal soldering amount. Figure 2 (b) in the figure is a mask constructed for a welding point with an excessive soldering amount. Figure 2 (c) in the figure is a mask constructed for a welding point with an insufficient soldering amount. Figure 2 (d) in the figure is a mask constructed for a welding point with a soldering tip. Specifically, for a welding point with a normal soldering amount, a mask with a downward concave is constructed; for a welding point with an excessive soldering amount, a mask with an upward convex is constructed; for a welding point with an insufficient soldering amount, a mask with a downward concave is constructed, but compared with a welding point with a normal soldering amount, the downward concave has a larger arc and a smaller area; for a welding point with a soldering tip, a mask with a protrusion on the contour is generated. In this way, the labeling and processing of the welding defect microscopic images are realized, and the real frame is obtained.

[0030] The welding defect microscopic images labeled and processed are subjected to data enhancement, and are converted into YOLO format to generate a welding point image data set.

[0031] Specifically, in order to increase the number of samples, the welding defect microscopic images labeled and processed are subjected to data enhancement, and the data is enhanced by three times by means of inversion and the like, and the JSON format labeling is converted into YOLO format (a labeling file format specially designed for target detection tasks), and finally a welding point image data set is generated.

[0032] S102: Construct a welding defect detection model based on an instance segmentation network model, and the welding defect detection model comprises a backbone network, a neck network and a detection head.

[0033] Specifically, the instance segmentation network model used in the present application is a YOLO11n-seg model.

[0034] S103: Replace the C3k2 module with the C2FRF module in the backbone network, optimize the input feature map, and input the optimized feature map to the neck network.

[0035] Replacing the C3k2 module with the C2FRF module in the backbone network, optimizing the input feature map, and inputting the optimized feature map to the neck network includes the following.

[0036] Specifically, the input feature map in this application is the feature map obtained by the convolution module from the solder defect microscopic image.

[0037] The C2FRF module is obtained by replacing the original Bottleneck module with the FRF module. The FRF module strengthens the information elements in the features by introducing the PConv operation. The specific operation process includes the following.

[0038] Specifically, the FRF module (Feature Refinement Feed-forward) is a feature refinement feed-forward module. The C3K2 module is a feature extraction module that combines variable convolution kernels and channel separation strategies. The Bottleneck module is a bottleneck module that combines convolution operations and optional residual connections.

[0039] The C2FRF module is the improved module name. After replacing the Bottleneck module with the FRF module based on the original structure, a new module is obtained. The PConv operation (Partial Convolution) is a partial convolution operation.

[0040] In the backbone network of YOLO11n-seg, to improve the detection performance of solder defects, the FRF module improves the traditional C3k2 module to obtain the C2FRF module. Since the traditional C3k2 module has weak recognition ability for various shapes of solder defects, the C2FRF module can effectively solve this problem. It can better preserve the detail information of the solder defects and improve the quality of the extracted features. By optimizing the redundant information, the model pays more attention to the key features and reduces the interference of background noise. At the same time, the C2FRF module cleverly combines the PConv operation, reduces the number of convolution kernels, simplifies the calculation process, reduces the processing amount of redundant information, and thus significantly reduces the computational complexity and memory consumption.

[0041] The input feature map is calculated by the PConv operation to reduce the redundant channel operation.

[0042] The first linear projection using 1x1 convolution adjusts the dimensions and enhances the nonlinearity through the GELU activation function to obtain intermediate features .

[0043] 1×1 convolution is a special convolution operation with a kernel size of 1×1.

[0044] obtained intermediate feature is expressed as: . Wherein, is a GELU activation function, is the first linear projection of 1×1 convolution, is a partial convolution operation, is the input feature map.

[0045] Split the intermediate feature along the channel into the first feature and the second feature , after Reshape transformation of the second feature , apply deep separable convolution operation to extract spatial features, and generate attention map through Flatten operation, and multiply the first feature element by element to realize feature modulation and obtain the intermediate feature after feature modulation.

[0046] Split the intermediate feature along the channel into the first feature and the second feature , the expression is: . Wherein, is a channel slicing operation.

[0047] The expression of the intermediate feature after feature modulation is: . Wherein, is the intermediate feature after feature modulation, is element-wise multiplication, is the Flatten operation, is the Reshape transformation, is the deep separable convolution.

[0048] The Flatten operation is a flattening operation in deep learning, and the purpose of the flattening operation is to convert a multi-dimensional tensor into a one-dimensional tensor. Reshape transformation is an operation that converts a tensor into a tensor with a specific shape, which adapts to the input requirements of different layers of the network.

[0049] The second linear projection of 1×1 convolution linearly transforms the intermediate feature after feature modulation, integrates the feature information of different dimensions, and outputs the optimized feature map .

[0050] Output the optimized feature map The expression is: wherein, is a second linear projection of the 1x1 convolution.

[0051] Specifically, the FRF module can extract representative features from the information flow, enhance feature representation, simplify redundant features, and reduce useless information features along the channel dimension. Therefore, replacing the original bottleneck module with the FRF module can better preserve the detailed information of the weld defect, thereby enhancing the detection accuracy of the model for the weld defect.

[0052] S104: In the neck network, the 2-5 level feature maps in the optimized feature map are introduced into the feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fusion feature map is obtained. The multi-level fusion feature map is input to the detection head.

[0053] The P2-FPN structure in the present application is a feature pyramid network introducing the P2 level.

[0054] In the neck network, the 2-5 level feature maps in the optimized feature map are introduced into the feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fusion feature map is obtained. The multi-level fusion feature map includes the following contents.

[0055] The nearest neighbor interpolation up-sampling is used to enlarge the size of the low-level feature map to be the same as the adjacent high-level feature Figure One .

[0056] The dynamic step size convolution down-sampling is used to reduce the size of the high-level feature map to be the same as the adjacent low-level feature Figure One .

[0057] The adjacent low-level feature map is channel spliced and sent into the C2FRF module to fuse the cross-stage features to obtain a multi-level fusion feature map.

[0058] For the P2, P3, P4 and P5 level feature maps extracted from the backbone network, firstly, P5 is spliced and fused with the down-sampled P4 to obtain N5, N5 is further fused by a C2FR module to obtain N'5 and is up-sampled and spliced with P4 and the down-sampled P3 to generate N4, N4 is further fused by a C2FRF module to obtain N'4 and is up-sampled and spliced with P3 and the down-sampled P2 to output N3; N3 is further fused by a C2FRF module to obtain N'3 and is spliced with the up-sampled N'4 to obtain M3, M3 is down-sampled by convolution and is spliced with N4, the down-sampled N'3 and the up-sampled N'5 to generate M4, M4 is down-sampled by convolution and is spliced with N'5 and the down-sampled N'4 to generate M5. In this way, the information in the feature maps of different levels is fully integrated, the detailed information in the low-level features and the semantic information in the high-level features are comprehensively utilized, and finally the multi-level fused feature maps M3, M4 and M5 are obtained.

[0059] The P2, P3, P4 and P5 levels are four levels of the backbone network. N5, N'5, N4, N'4, N3 and N'3 are all intermediate feature maps in the top-down fusion path.

[0060] The feature maps of adjacent low levels are spliced in the channel and sent into the C2FRF module to fuse the cross-stage features, so as to obtain the multi-level fused feature maps.

[0061] In the above manner, the information in the feature maps of different levels is fully integrated, the detailed information in the low-level features and the semantic information in the high-level features are comprehensively utilized, and finally the multi-level fused feature maps are obtained.

[0062] The PAFPN structure (path aggregation feature pyramid network structure) used in the original YOLO11n-seg model tends to combine feature maps of the same scale, and has insufficient processing of multi-scale information, lacking comprehensive processing and fusion of multi-scale information from different resolution layers, which weakens the learning ability of the model for small and unevenly sized weld defects. The P2-FPN structure of the present application effectively solves this problem. The low-level features introduced in the present application have relatively weak semantic information, but contain more weld edge information and detail information, retain higher resolution spatial information, and can capture the small irregularities of the weld edge, which is crucial for accurate detection of weld defects. The present application continues to use the C2FRF module to effectively fuse multi-scale features in the P2-FPN structure, while adding P2-level feature maps to form P2-FPN, forming a four-level pyramid of P2-P5, making the feature transmission from shallow to deep more continuous. Under a microscope, welds are often accompanied by high reflectivity, metal particle noise or PCB background texture interference, and high-level features are easily contaminated by semantic information, while low-level features contain more bottom-level gradient information, which can better distinguish real defects from reflective artifacts, thereby improving the detection accuracy of the model in complex backgrounds.

[0063] S105: In the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to convert the multi-level fused feature map into a weld defect detection result.

[0064] In the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to convert the multi-level fused feature map into a weld defect detection result, including the following contents.

[0065] The parameter sharing method is a design in which the multi-level fused feature map shares the same detection head parameters.

[0066] A depth separable convolution and a CBS module are used to construct a lightweight detection head to convert the multi-level fused feature map into a weld defect detection result.

[0067] Specifically, the parameter sharing method breaks the traditional mode of setting independent detection head parameters for different level feature maps. By sharing parameters, the number of parameters of the detection head as a whole is reduced, the complexity of the model is reduced, and at the same time it is also helpful for the model to extract more common feature representations in the learning process of different level features, improving the generalization ability of the model. A depth separable convolution (DWConv) and a CBS module (convolution-batch normalization-activation module) are used to construct a lightweight detection head to achieve the goal of converting the multi-level fused feature map into a weld defect detection result.

[0068] Figure 3A schematic diagram of an improved detection head for the original YOLO11n-seg model provided in an embodiment of the present application. To further enhance the lightweight nature of the detection head, a DWConv convolution module (depthwise separable convolution) with a 3×3 convolution kernel and a CBS module with a 1×1 convolution kernel are used to replace the original convolution module. The DWConv convolution decomposes the standard convolution into depthwise and pointwise convolutions, significantly reducing the amount of computation and the number of parameters. The CBS module, through a combination of convolution, batch normalization, and activation functions, further optimizes computational efficiency while maintaining feature extraction capabilities. Figure 3 The DW-Head in the figure is the detection head.

[0069] The detection head is responsible for converting the feature maps extracted by the backbone network and fused by the neck network into the final detection results, covering tasks such as classification and regression. Traditional detection heads usually require a large number of parameters, which not only increases the computational cost, but may also affect the application of the model in resource-constrained scenarios. This application adopts a parameter-sharing detection head method and uses DWConv convolution to construct the detection head, aiming to achieve lightweight detection heads, while avoiding accuracy loss as much as possible, to meet the actual needs of high-resolution microscopic image processing and edge deployment.

[0070] S106: Using a joint loss function including a bounding box regression loss function based on a dynamic focusing mechanism and a focus loss function to guide the training of the solder joint defect detection model.

[0071] Specifically, in the training of the solder joint defect detection model, in order to deal with the problems of complex noise interference in the microscopic images of solder joint defects, difficulty in obtaining defect samples, imbalance of sample types (relatively few negative samples), etc., this application adopts a joint loss function including a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function to guide the training of the solder joint defect detection model.

[0072] The expression of the bounding box regression loss function based on the dynamic focusing mechanism is: .in, is the bounding box regression loss based on the dynamic focusing mechanism, is the non-monotonic focusing coefficient, , and is the hyperparameter used to calculate the non-monotonic focusing coefficient, is the outlier degree of the anchor box, , is the gradient gain, for The exponential moving average of It is a bounding box regression loss function built based on IoU, where IoU is an evaluation indicator that measures the degree of overlap between the predicted box and the true box. For penalty items. and The value of the can be adjusted according to different data sets, and in the present application is taken as 1.9, is taken as 3, the highest detection accuracy can be achieved.

[0073] , wherein, is an exponential calculation, is the horizontal coordinate of the prediction box, is the vertical coordinate of the prediction box, is the horizontal coordinate of the real box, is the vertical coordinate of the real box, is the width of the smallest closed box formed by the prediction box and the real box, is the height of the smallest closed box formed by the prediction box and the real box, is the result of separating from the calculation graph.

[0074] The calculation graph is a basic concept in deep learning, that is, a data structure used for automatic derivation in the loss function calculation process.

[0075] Specifically, the bounding box regression loss function based on the dynamic focusing mechanism is mainly used to solve the problem of low data quality. By introducing the outlying degree evaluation anchor box quality, the detection head is effectively focused on the anchor box of ordinary quality, and the detection performance is improved.

[0076] The expression of the focal loss function is: , wherein, is the focal loss function, is the class balance coefficient, is the matching degree of the prediction probability and the real label, is the weight of the difficult sample.

[0077] Specifically, the class balance coefficient can be set according to the characteristics of the data set. The class balance coefficient is used to adjust the contribution proportion of positive and negative samples to the total loss, The greater the set, the smaller the influence of the negative sample on the loss. Since the proportion of positive samples in the solder joint image data set constructed in the present application is small and more difficult to detect, the weight of the difficult sample is introduced on the basis of to control the attention degree of the model to the difficult sample. The value range of is 0-1, The value range of is 0-5. By continuously experimenting to select the appropriate value, the parameter setting used in the present application is , .

[0078] Specifically, the focal loss function is mainly used to solve the class imbalance problem, by reducing the loss of easy-to-classify samples (positive samples), so that the model pays more attention to difficult-to-classify (negative samples) samples.

[0079] The expression of the joint loss function is: . Wherein, is the joint loss function, is a hyperparameter, with a value range of 0 to 1, used to adjust the balance between the bounding box regression loss function based on the dynamic focusing mechanism and the focal loss function to achieve the optimal synergy of the two loss functions. The hyperparameter can be set to 0.4.

[0080] Specifically, the joint loss function combines the bounding box regression loss function based on the dynamic focusing mechanism and the focal loss function, aiming to simultaneously consider the industrial noise problem and the defect class imbalance problem of micro-welding spot images, and improve the detection performance of the model in complex industrial environments.

[0081] S107: Input the welding spot image dataset into the welding spot defect detection model for training, to obtain a model weight file.

[0082] Specifically, the present application is based on the Pytorch1.13.0+cuda11.6 deep learning framework for model training. The present application uses Windows 11 operating system. The GPU selected is NVIDIA GeForce RTX 4060, which provides powerful computing power for model training. The CPU is i7-137000T, which guarantees the overall running efficiency of the system. The model development is carried out using Python 3.10 programming language.

[0083] The training parameters of the present application are set as follows. The model input size is set to 640x640, ensuring that the input images have uniform specifications for easy model processing. The number of samples per batch is set to 16, i.e. 16 images are processed each time, balancing the calculation efficiency and training effect. The stochastic gradient descent method (SGD) is used to optimize the network parameters, with a momentum of 0.937, which helps to accelerate the convergence of the model and avoid falling into local optimal solution. The initial learning rate of the weight is 0.01, which provides a suitable step size for model parameter update; the weight decay is 0.0005, which prevents model overfitting. A total of 200 rounds of training are performed, while setting an early stopping mechanism, which stops training in advance when the model performance on the validation set no longer improves, saving training time and computing resources.

[0084] The processing procedure of the backbone network is as follows. After the solder joint image dataset is input into the backbone network of the solder joint defect detection model, the image is processed in turn through the convolution module and the designed C2FRF module. In the feature extraction process, important features are strengthened, redundant information is simplified, useless features are removed, and the detail information of the solder joint defect is retained, and finally the optimized feature map is output. The optimized feature map contains different levels of feature information extracted from the solder joint defect image, providing a basis for the processing of the subsequent modules.

[0085] The processing procedure of the neck network is as follows. The neck network receives the optimized feature map from the backbone network and the feature information of different sizes captured by the C2FRF module. Further, the P2-FPN structure is used to comprehensively process and fuse the multi-scale detail features of different resolution layers, effectively combining high-level semantic information with low-level spatial information, and outputting a multi-level fusion feature map to improve the detection capability of the model for defects of different scales.

[0086] The processing procedure of the detection head is as follows. The detection head receives the multi-level fusion feature map processed by the neck network. In the detection head, a parameter sharing method is used for lightweight defect detection classification of the multi-level fusion feature map, which reduces the computational complexity of the model while ensuring detection accuracy and improves detection efficiency.

[0087] S108: A microscope is used to shoot a solder joint defect microscopic image to be detected, and a model weight file is used to test the solder joint defect microscopic image to be detected, to obtain a solder joint defect detection result.

[0088] Specifically, the model performs feature extraction, fusion and classification on the input solder joint defect microscopic image to be detected, and finally obtains a solder joint defect detection result, realizing accurate identification and classification of PCB solder joint defects.

[0089] Figure 4 The precision-recall curve of the original YOLO11n-seg model provided by the embodiments of the present application is shown in the figure. Figure 5 The precision-recall curve of the solder joint defect detection model provided by the embodiments of the present application is shown in the figure. Figure 4 and Figure 5 In the figures, the horizontal axis Recall represents recall rate, and the vertical axis Precision represents precision. good represents a normal soldering amount category, Figure 4 0.947, Figure 5 0.972. exc_solder represents a soldering amount category, Figure 4 0.980, Figure 5 0.988. poor_solder represents a soldering amount category, Figure 4 0.908, Figure 5is 0.956. Spike represents the category of solder tip, Figure 4 is 0.770, Figure 5 is 0.823. All classes represents the average precision (mAP@0.5) of all categories, that is, the average precision when the IoU threshold is 0.5, Figure 4 is 0.901, Figure 5 is 0.935. The values of the solder defect detection model of the present application in each category are improved, proving the effectiveness of the improved model.

[0090] Figure 6 is a solder defect detection result visualization diagram of the solder defect detection model provided by the embodiment of the present application. From Figure 6 It can be seen that the solder shape is complex and the size is different. Since the C2FRF and P2-FPN modules are designed in the solder defect detection model, the network's ability to capture global information and suppress noise is improved, and different layer receptive field features are effectively fused, so the solder defects with different sizes and blurred boundaries can be accurately detected, and the detection accuracy of the original model is improved.

[0091] In summary, based on the solder defect detection model, different types of solder defects of different sizes can be well detected. The improved model incorporates the P2-FPN module, which can effectively solve the problem of difficult detection of multi-scale solder microscopic images. By incorporating the P2 layer into feature fusion, effective fusion of high-level semantic features and low-level features is achieved, significantly improving the feature expression ability of the model. The parameter-shared DW-head detection head and the PConv and DWConv introduced in the C2FRF significantly reduce the computational complexity and memory consumption. At the same time, the improved joint loss function mixes two loss functions, which not only enhances the robustness of the model, but also does not affect the model parameter amount and other indicators.

[0092] The embodiment of the present application also provides a PCB solder defect detection device 700 based on a microscopic image, as shown in the figure, the device comprises: a processing module 701, a construction module 702, a guidance module 703 and a training module 704. Figure 7

[0093] The processing module 701 is used to acquire solder defect microscopic images containing global three-dimensional morphology by using a microscope, and to perform preprocessing and labeling processing on the acquired solder defect microscopic images to generate a solder image dataset.

[0094] ​The construction module 702 is configured to construct a welding defect detection model based on an instance segmentation network model, the welding defect detection model comprising a backbone network, a neck network and a detection head. In the backbone network, a C2FRF module is used to replace a C3k2 module, and the input feature map is optimized, and the optimized feature map is input to the neck network. In the neck network, the 2-5 level feature maps in the optimized feature map are introduced into a feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fusion feature map is obtained, and the multi-level fusion feature map is input to the detection head. In the detection head, a parameter sharing method and a depth separable convolution lightweight detection head are used to convert the multi-level fusion feature map into a welding defect detection result.

[0095] The guidance module 703 is configured to guide the welding defect detection model training by using a joint loss function comprising a boundary box regression loss function based on a dynamic focus mechanism and a focal loss function.

[0096] The training module 704 is configured to input a welding image dataset into the welding defect detection model for training to obtain a model weight file, use a microscope to capture a welding defect microscopic image to be detected, and use the model weight file to test the welding defect microscopic image to be detected to obtain a welding defect detection result.

[0097] Some of the modules in the device described in the present application can be described in the general context of computer-executable instructions, such as program modules, which are executed by computers. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like, which perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0098] The devices or modules described in the above application embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described as various modules with functions and are described respectively. In the implementation of the embodiments of the present application, the functions of each module can be implemented in the same software and / or hardware or multiple software and / or hardware. Of course, the modules implementing certain functions can also be implemented by multiple sub-modules or sub-units.

[0099] The methods, apparatuses or modules described in the present application can be implemented in a computer readable program code manner. The controller can be implemented in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (for example, software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (Application Specific Integrated Circuit, in short: ASIC), a programmable logic controller and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in a pure computer readable program code manner, the same function can also be implemented by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0100] As shown in Figure 8 The embodiment of the present application also provides a PCB solder joint defect detection server based on a microscopic image, which comprises a memory 801 and a processor 802; the memory 801 is used for storing computer executable instructions; and the processor 802 is used for executing the computer executable instructions to realize the PCB solder joint defect detection method based on a microscopic image provided in the above embodiment of the present application.

[0101] The embodiment of the present application also provides a computer readable storage medium, which stores executable instructions, and a computer executes the executable instructions to realize the PCB solder joint defect detection method based on a microscopic image provided in the above embodiment of the present application.

[0102] From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.

[0103] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment mainly describes the difference from other embodiments. The whole or part of the present application can be used in a plurality of general or special computer system environments or configurations.

[0104] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A microscopic image-based PCB solder joint defect detection method, characterized in that, The method comprises the steps of: obtaining a micro image of a welding point defect containing global stereoscopic morphology by using a microscope, preprocessing and labeling the obtained micro image of the welding point defect, and generating a welding point image dataset; constructing a welding point defect detection model based on an instance segmentation network model, wherein the welding point defect detection model comprises a backbone network, a neck network and a detection head; replacing a C3k2 module with a C2FRF module in the backbone network to optimize the input feature map, and inputting the optimized feature map into the neck network; in the neck network, introducing the 2-5 level feature maps in the optimized feature map into a feature pyramid to form a P2-FPN structure for feature fusion, and obtaining a multi-level fusion feature map, and inputting the multi-level fusion feature map into the detection head; in the detection head, using a parameter sharing method and a depth separable convolution lightweight detection head to convert the multi-level fusion feature map into a welding point defect detection result; using a joint loss function comprising a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function to guide the training of the welding point defect detection model; inputting the welding point image dataset into the welding point defect detection model for training to obtain a model weight file; using a microscope to take a micro image of a welding point defect to be detected, and using the model weight file to test the micro image of the welding point defect to be detected to obtain a welding point defect detection result.

2. The microscopic image-based PCB solder joint defect detection method according to claim 1, characterized in that, The preprocessing and labeling of the obtained micro image of the welding point defect to generate the welding point image dataset comprises: using a method combining contrast limited adaptive histogram equalization and bilateral filtering to preprocess the obtained micro image of the welding point defect; dividing the welding point morphology in the micro image of the welding point defect into four categories of normal soldering tin amount, excessive soldering tin, insufficient soldering tin and soldering tin tip, and obtaining a real label; polygonally labeling the welding points, creating masks with different contours according to the categories of each welding point, labeling the obtained micro image of the welding point defect, and obtaining a real box; performing data enhancement on the micro image of the welding point defect after labeling, and converting it into YOLO format to generate the welding point image dataset.

3. The microscopic image-based PCB solder joint defect detection method of claim 2, wherein, The C2FRF module is used to replace the C3k2 module in the backbone network to optimize the input feature map, and the optimized feature map is input into the neck network, which comprises: replacing the original Bottleneck module with an FRF module to obtain a C2FRF module, and the FRF module strengthens the information elements in the features by introducing a PConv operation, and the specific operation process comprises: performing a PConv operation on the input feature map reducing redundant channel operations; First linear projection using 1x1 convolution Adjust the dimension and enhance the nonlinearity by the GELU activation function to get the intermediate feature ; Split the intermediate feature into a first feature and a second feature , and apply a Reshape transformation to the second feature , then extract spatial features by applying a depth separable convolution operation, and generate an attention map by a Flatten operation, and perform element-wise multiplication with the first feature to achieve feature modulation, and obtain the intermediate feature after feature modulation; Second linear projection by 1x1 convolution Linear transformation is performed on the intermediate features after feature modulation, the feature information in different dimensions is integrated, and the optimized feature map is output after GELU activation function activation .

4. The microscopic image-based PCB solder joint defect detection method according to claim 3, characterized in that, In the neck network, the 2-5 level feature maps in the optimized feature map are introduced into a feature pyramid to form a P2-FPN structure for feature fusion to obtain a multi-level fusion feature map, which comprises: using nearest neighbor interpolation upsampling to enlarge the size of the low-level feature map to be consistent with the adjacent high-level feature map; using a dynamically adjusted step convolution downsampling to reduce the size of the high-level feature map to be consistent with the adjacent low-level feature map; concatenating the adjacent low-level feature maps in the channel and sending them into the C2FRF module to fuse the cross-stage features to obtain a multi-level fusion feature map.

5. The microscopic image-based PCB solder joint defect detection method of claim 4, wherein, The parameter sharing mode is a design of sharing the same detection head parameter of the multi-level fusion feature map. The parameter sharing mode is a design of sharing the same detection head parameter of the multi-level fusion feature map. The parameter sharing mode is a design of sharing the same detection head parameter of the multi-level fusion feature map.

6. The microscopic image-based PCB solder joint defect detection method according to claim 5, wherein, The joint loss function includes a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function. The expression of the bounding box regression loss function based on the dynamic focusing mechanism is: ; wherein, is the bounding box regression loss based on the dynamic focusing mechanism, is a non-monotonic focusing coefficient, , and are hyperparameters for calculating the non-monotonic focusing coefficient, is the outlying degree of the anchor box, , is the gradient gain, is the exponential moving average of , is the bounding box regression loss function constructed based on IoU, wherein IoU is an evaluation index for measuring the degree of overlap between the predicted box and the true box, is a penalty term; ; wherein, is an exponential calculation, is a horizontal coordinate of the prediction box, is a vertical coordinate of the prediction box, is a horizontal coordinate of the ground truth box, is a vertical coordinate of the ground truth box, is a width of the smallest enclosing box formed by the prediction box and the ground truth box, is a height of the smallest enclosing box formed by the prediction box and the ground truth box, is a result of separating from the computation graph; The expression of the focal loss function is: ; wherein, is the focal loss function, is a category balancing coefficient, is the matching degree of the prediction probability and the true label, is the weight of the difficult sample; The expression of the joint loss function is: ; wherein, is the joint loss function, is a hyperparameter.

7. A microscopic image-based PCB solder joint defect detection apparatus, characterized by, The joint loss function includes a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function. The processing module is configured to acquire a solder defect microscopic image containing a global three-dimensional morphology by using a microscope, and to generate a solder image dataset by preprocessing and labeling the acquired solder defect microscopic image. The construction module is configured to construct a solder defect detection model based on an instance segmentation network model, wherein the solder defect detection model includes a backbone network, a neck network, and a detection head. In the neck network, the 2th to 5th level feature maps in the optimized feature maps are introduced into a feature pyramid to form a P2-FPN structure for feature fusion, and a multi-level fusion feature map is obtained. The joint loss function includes a bounding box regression loss function based on a dynamic focusing mechanism and a focal loss function. The training module is configured to input the solder image dataset into the solder defect detection model for training to obtain a model weight file.

8. A microscopic image-based PCB solder joint defect detection server, characterized by, The training module is configured to input the solder image dataset into the solder defect detection model for training to obtain a model weight file. The memory is configured to store computer executable instructions. The processor is configured to execute the computer executable instructions to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores executable instructions, and the computer executes the executable instructions to implement the method of any one of claims 1-6. The computer readable storage medium stores executable instructions, and the computer executes the executable instructions to implement the method of any one of claims 1-6.

Images