Multi-model fusion PCB through-hole defect detection method, device, equipment and medium

Abstract

The present invention discloses a multi-model fusion PCB through-hole defect detection method, device, equipment and medium. The method includes: obtaining a sample image set to be processed, and processing the sample image set to obtain a target sample image set; pruning and training each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; the model structure of each deep learning model to be trained is different from each other; establishing a target fusion model based on the target deep learning model, and inputting the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. By fusing multiple models and combining pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved while ensuring high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-model fusion PCB through-hole defect detection method, device, equipment and medium. Background Art

[0002] With the rapid development of the electronics manufacturing industry, printed circuit boards (PCBs), the foundation and core components of electronic products, have become increasingly important for their performance and reliability. PCB vias, critical structures used to connect different layers within a circuit board, are widely used in the manufacture of multilayer circuit boards. During the PCB production process, vias can develop defects for a variety of reasons, such as uneven diameters, cracks in the hole walls, and copper foil shedding. These defects directly impact the electrical performance and reliability of the circuit and may even cause product failure.

[0003] However, traditional through-hole defect detection methods mostly rely on manual visual inspection and traditional optical inspection equipment, which have shortcomings such as low efficiency and significant influence from human factors. Summary of the Invention

[0004] The present invention provides a multi-model fusion PCB through-hole defect detection method, device, equipment and medium. By fusing multiple models and combining them with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, meeting real-time requirements, and solving the problem of reduced model inference speed in the existing technology.

[0005] According to one aspect of the present invention, a multi-model fusion PCB through-hole defect detection method is provided, which includes:

[0006] Acquire a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set;

[0007] Performing pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model, and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other;

[0008] A target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected.

[0009] According to another aspect of the present invention, a multi-model fusion PCB through-hole defect detection device is provided, which includes:

[0010] A sample image determination module is used to obtain a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set;

[0011] a model training module, configured to perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model, and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other;

[0012] A defect detection module is used to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the through-hole defect detection result corresponding to the image to be detected.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and,

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the PCB through-hole defect detection method of multi-model fusion described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-model fusion PCB through-hole defect detection method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention obtains a set of sample images to be processed, processes the set of sample images to be processed to obtain a target sample image set, and prunes and trains each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained. Finally, a target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, meeting real-time requirements.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flowchart of a multi-model fusion PCB through-hole defect detection method provided according to an embodiment of the present invention.

[0022] Figure 2 4 is a flowchart of a key point detection model provided according to an embodiment of the present invention.

[0023] Figure 3 2 is a schematic diagram of a channel scaling factor provided according to an embodiment of the present invention.

[0024] Figure 4 This is a flowchart of a multi-model fusion PCB through-hole defect detection method provided according to an embodiment of the present invention.

[0025] Figure 5 3 is a schematic diagram of a target fusion model processing process provided according to an embodiment of the present invention.

[0026] Figure 6 3 is a schematic diagram of the information fusion module processing process provided according to an embodiment of the present invention.

[0027] Figure 7 It is a schematic diagram of the processing process of the prompt information interaction module provided according to an embodiment of the present invention.

[0028] Figure 8 It is a structural schematic diagram of a multi-model fusion PCB through-hole defect detection device provided according to an embodiment of the present invention.

[0029] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1:

[0033] Figure 1 The present invention provides a flowchart of a multi-model fusion PCB through-hole defect detection method. This embodiment is applicable to the case where, during the PCB production process, the through-hole defect of the PCB is automatically detected based on the collected PCB image. The method can be executed by a multi-model fusion PCB through-hole defect detection device. The multi-model fusion PCB through-hole defect detection device can be implemented in the form of hardware and / or software. The multi-model fusion PCB through-hole defect detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110 , obtaining a sample image set to be processed, and processing the sample image set to be processed to obtain a target sample image set.

[0035] The sample image set to be processed may be an unprocessed original image obtained after scanning the PCB, and the target sample image set may be understood as a set of sample images obtained after processing the original image.

[0036] Specifically, the PCB is scanned by an X-ray device to obtain an unprocessed PCB image, and the sample image set to be processed is processed to obtain a target sample image set. For example, the images in the sample image set to be processed can be cleaned and enhanced to obtain the target sample image set.

[0037] On the basis of the above technical solution, the processing of the sample image set to be processed to obtain the target sample image set includes: annotating the sample image set to be processed based on image annotation software to obtain a sample image set containing annotated labels; dividing the sample image set into a sample image set and a test image set according to a preset ratio, and determining the target sample image set based on the sample image set and the test image set.

[0038] The image annotation software may be software for annotating images to be annotated, such as Labelme software. The annotation labels may be pre-set image labels, including key point labels, object detection labels, and grayscale labels. The preset ratio may be understood as a pre-set ratio for dividing the sample set. The sample image set may be a set of images used to train the model, and correspondingly, the test image set may be a set of images used to test the model.

[0039] Specifically, select appropriate image annotation software according to needs, such as LabelImg, LabelMe, VOTT and other annotation software. The annotation labels include three categories: key points, target detection and mask. Then use rotating annotation software to annotate the images in the sample image set to be processed, and after annotation, divide the sample image set into a sample image set and a test image set according to a preset ratio. For example, it can be divided into a training set and a test set at a ratio of 7:3.

[0040] On the basis of the above technical solution, the target sample image set is determined based on the sample image set and the test image set, including: adjusting the image sizes in the sample image set and the test image set based on a preset image size, and performing random horizontal flipping processing; converting the images subjected to random horizontal flipping processing into a tensor format, and performing normalization processing to obtain the target sample image set.

[0041] The preset image size can be understood as the size of the input image that is preset. The tensor format can be a format used to describe the image as a tensor.

[0042] Specifically, the data is preprocessed, the input image size is changed to 1024×1024, and then randomly flipped horizontally with a probability of 0.5. After the image is converted to Tensor format, it is standardized and normalized. For example, with the help of the Python language environment, the torchvision.Resize() method is used to adjust the image size. Subsequently, the torchvision.RandomCrop() method is applied to randomly crop the image, and then the torchvision.RandomHorizontalFlip() method is used to perform a horizontal random flip operation on the image with a probability of 50%. Next, the processed image is converted to Tensor format through the transforms.ToTensor() method, and preliminary normalization processing is implemented in the process. Finally, the transforms.Normalize() function is used to complete the standardization processing step of the image data to ensure that the data meets the predefined normalization and standardization requirements. The specific algorithm details of the standardization and normalization include: ; Among them, μ represents the mean of the image, x represents the tensor of the image, σ represents the standard deviation, x max Indicates the maximum value of the image tensor, x min Represents the minimum value of the image tensor, Represents the normalized image tensor, Represents the normalized image tensor.

[0043] S120. Perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained.

[0044] The deep learning model to be trained may be an initialized deep learning model, and may include a key point model to be trained, a detection model to be trained, and a segmentation model to be trained. The target deep learning model includes a target key point model, a target detection model, and a target segmentation model; and the model structures of the deep learning models to be trained are different from each other.

[0045] Specifically, we establish key point models, detection models, and segmentation models, using LiteHRnet, YOLOV8, and Unet models respectively. We then initialize the key point models, detection models, and segmentation models, design a structured pruning training framework, and perform pruning training on the key point models, detection models, and segmentation models respectively. Figure 2As shown, since the size of the through-hole accounts for a very small proportion of the entire image, an additional decoding head is added to the output layer of the LiteHRnet model, and an additional resolution map is output through the decoding head. Deconvolution is used in the additional decoding head, and finally a HeatMap of the original size is output. Since the key point detection model is only used to locate the target position, it can be understood as a multi-target single-type key point detection model, so the detection logic of the bottom-up framework is adopted. Referring to the CenterNet model, the output of LiteHRnet is the Heatmap feature map, and the training labels are the Heatmap feature map and mask. The mask is generated according to the coordinates corresponding to the maximum value of each Gaussian kernel in the Heatmap label. The FocalLoss loss function is used to calculate the Heatmap feature loss. Assuming that the input image is , the output Heatmap is , FocalLoss can be expressed as:

[0046] ;in, and are hyperparameters and are set to 2 and 4 respectively.

[0047] YOLOv8 uses the lightest version YOLOV8-s, whose label consists of three parts: bounding box parameters, category labels and confidence. Based on gridcell, each grid predicts multiple bounding boxes. The label of each bounding box includes: the center point of the bounding box, the width and height of the bounding box, and the probability distribution of the confidence and category. Its form is YOLOv8's loss function consists of multiple parts, including: LocalizationLoss: used to measure the difference between the predicted bounding box and the true bounding box. Usually IoU (Intersection over Union) and MSE (Mean Squared Error) are used to calculate: ;

[0048] Confidence Loss is used to measure the difference between the predicted confidence and the actual confidence. Binary crossentropy (BCE) is often used to calculate it: ;

[0049] ClassificationLoss is used to measure the difference between the predicted category and the true category, and is usually calculated using cross-entropy loss: ; Then we can determine the final loss function combination as:

[0050] ;

[0051] The encoder of the segmentation model Unet uses ResNet50. The label is used to mark the region of the through-hole position. The region of the through-hole position is extracted using a binary classification method. The loss function is:

[0052] .

[0053] On the basis of the above technical solution, the pruning training of each deep learning model to be trained based on the target sample image set includes: determining the scaling factor of each channel corresponding to the batch normalization layer in each deep learning model to be trained; regularizing the scaling factor of each channel, and pruning the channels of the batch normalization layer according to the regularization processing result.

[0054] Among them, the batch normalization layer can be used to standardize the input of each layer of the neural network, which can solve the distribution drift of input data during training.

[0055] Specifically, in the deep learning model, find all batch normalization layers, and for each batch normalization layer, extract its corresponding scaling factor. It should be noted that the scaling factor determines the importance of each channel in the feature map, and then during the model training process, the regularization term is added to the loss function. According to the scaling factor after regularization, the pruning threshold is determined, and channels below the pruning threshold will be pruned. For example, in deep learning, the BatchNormalization (BN) layer is a commonly used technology to accelerate training and improve the generalization ability of the model. The core operation of the batch normalization layer is to standardize the input of each layer to ensure that its mean is 0 and the variance is 1. The batch normalization layer usually includes two important learnable parameters: the scaling factor (scalefactor) and offset factor .

[0056] In order to achieve batch normalization layer pruning, these parameters are regularized to reduce unimportant channels and weights. Batch normalization layer pruning mainly selects the channels that need to be pruned by evaluating the importance of each channel. Through L1 regularization, the model can automatically reduce the weights of unimportant channels. Parameters are compressed to zero, thus achieving channel pruning. For an input tensor , the operation process of the batch normalization layer can be expressed by the following formula: ;in, is input The mean over the batch dimension, is the variance of the input, is a small constant used to prevent the denominator from being zero. The batch normalization layer then scales and offsets the normalized input: ;in, is the scaling factor for each channel, is the offset factor for each channel.

[0057] The goal of batch normalization layer pruning is to regularize the scaling factor , remove channels that contribute less to the model. The specific approach is to evaluate the importance of each channel through L1 regularization and The value of determines which channels can be pruned. Channels with less importance will be pruned, thereby reducing the amount of computation and the number of parameters of the network. Figure 3 For each channel in the batch normalization layer , whose scaling factor Its importance can be assessed by adding L1 regularization. The goal of L1 regularization is to make The weights of channels with smaller values tend to be zero. The L1 regularization term can be expressed as: ; The L1 regularization term is applied to the loss function through the gradient descent optimization process, which promotes some unimportant is shrunk or becomes zero. The total loss function becomes: .

[0058] On the basis of the above technical solution, after performing channel pruning on the batch normalization layer according to the regularization processing results, it also includes: determining the optimal reconstruction matrix and the optimal model parameters based on minimizing the reconstruction error function; and deploying the feature reconstruction layer after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters.

[0059] The minimization of the reconstruction error function can be used to determine the loss function of the feature reconstruction layer. The reconstruction matrix is used to convert the pruned feature maps back to a form close to the original feature maps, thereby reducing the information loss caused by pruning. The feature reconstruction layer has a structure of three convolutional layers in series.

[0060] Specifically, a reconstruction error function is constructed. The reconstruction error function can measure the difference between the output of the pruned model and the output of the original model, and then the reconstruction matrix is solved by an optimization algorithm (such as gradient descent, least squares method, etc.) to minimize the reconstruction error function. While solving the reconstruction matrix, the parameters of the model also need to be updated to ensure that the model can still maintain good performance after pruning and reconstruction. Then, a feature reconstruction layer is designed according to the optimal reconstruction matrix. This layer will be applied after the batch normalization layer after pruning. It should be noted that in the process of channel pruning, since some information is removed, it will affect the accuracy of the model. Therefore, the feature map reconstruction step aims to restore the lost feature information to ensure the performance of the model. For some information lost after pruning, a reconstruction network is designed to fit the feature maps before and after pruning to restore the lost information. Assume that the original feature map of the output of the network before pruning is , after pruning, the feature map obtained , , some channels are pruned. Among them, b represents the batch dimension (batch), which is used to indicate the number of samples input at a time. During the training process, inputting multiple samples at a time can improve computational efficiency. c: represents the channel dimension (channel), which is used to indicate the number of channels of the input data or the number of feature maps. For convolutional neural networks (CNN), it usually indicates the number of output feature maps of the convolutional layer. h: represents the height dimension (height), which is used to indicate the height of the image or feature map. w: represents the width dimension (width), which is used to indicate the width of the image or feature map.

[0061] Hope that by the pruned feature map Reconstruct to get a feature map as close to the original as possible The feature map of the pruned feature map Input to the reconstruction network Ref. Usually, it is expected that the pruned features have similar output features to the original model after the reconstruction model:

[0062] ;

[0063] in, Represents the error term, which represents the part that cannot be completely restored by linear combination, For the reconstruction module, the pruned feature map structure is transformed to be the same as the original feature map structure. It should be noted that the feature reconstruction layer is a three-layer convolution structure, using a 1×1 convolution, a 3×3 convolution and a 1×1 convolution structure to transform the pruned feature map. The spatial structure of The sum of squares of , i.e. minimizing the reconstruction error: ; among them, among them represents the Frobenius norm, which is the square root of the sum of the squares of the elements of the matrix. Represents the network parameters before pruning, and the optimal reconstruction matrix is obtained by minimizing the above loss function and model parameters After completing the training of the reconstructed network, it is usually necessary to fine-tune the pruned network to ensure that the fusion of the reconstructed network and the main network does not affect the model performance.

[0064] S130. Establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the through-hole defect detection result corresponding to the image to be detected.

[0065] The target fusion model can be understood as a deep learning model obtained by combining the target deep learning models. The image to be inspected can be a PCB image that requires through-hole defect inspection.

[0066] Specifically, after the target deep learning model training is completed, a target fusion model can be constructed according to each target deep learning model, and the image to be detected is input into the target fusion model to obtain the through-hole defect detection result output by the target fusion model.

[0067] The technical solution of the embodiment of the present invention obtains a set of sample images to be processed, processes the set of sample images to be processed to obtain a target sample image set, and prunes and trains each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained. Finally, a target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, meeting real-time requirements.

[0068] Example 2:

[0069] Figure 4 This is a flowchart of a multi-model fusion PCB through-hole defect detection method provided by an embodiment of the present invention. This embodiment further refines the technical solution of establishing a target fusion model based on the target deep learning model on the basis of the above technical solution. Figure 4 As shown, the method includes:

[0070] S210, removing the output layer in the target key point model and the target segmentation model, and inputting the hidden layer results in the target key point model and the target segmentation model into a prompt information fusion module to obtain a fused prompt feature output by the prompt information fusion module.

[0071] The hidden layer result may be the output result of the last hidden layer in the key point model and the segmentation model. The information fusion module may be understood as a processing model for fusing the output features of the target key point model and the target segmentation model.

[0072] Specifically, the entire fusion structure of the fusion model is as follows Figure 5 As shown in the figure, the key point detection model and the segmentation model load the pruned pre-trained parameters, remove the output layer, and use the last hidden layer features as the output, which are input into the Prompt fusion module to generate a multimodal prompt prior. During the training process, the key point detection model and the segmentation model will freeze the parameters and will not participate in the network parameter update. As a multimodal feature generator, the key point information and the mask information are fused through the Prompt fusion module. As a multimodal prompt feature, it is fused with the multi-level output features of the YOLOV8 Backbone through the Prompt interaction module. The design goal of the Prompt interaction module is to establish an effective connection between different modal features so that the key point information and the mask information can provide more accurate context information for the YOLOv8 detection model, thereby enhancing the model's recognition ability in through-hole defect detection.

[0073] It should be noted that the structure of the Prompt fusion module is as follows Figure 6As shown in the figure, because the output features of the segmentation model and the keypoint model are misaligned in both spatial and channel dimensions, a 1×1 convolution is first applied to both features to ensure they have the same number of channels. Next, a 3×3 convolution block is used to enhance the features, aiming to extract richer feature information and improve their representational capabilities. The enhanced features are spatially scaled by adjusting their spatial resolution through a spatial transformation, aligning the spatial scales of the two features. However, spatial transformation can introduce interference and unwanted noise, so another 3×3 convolution block is used to suppress this interference, ensuring feature quality and effectiveness. To avoid suboptimal results that may result from static fusion, a channel-wise attention mechanism is introduced to dynamically adjust the importance of the fused features. Before feature concatenation, the input features first undergo a global average pooling operation to obtain global information, then undergo a 1×1 convolution to reduce the channel-wise dimension, and finally generate dynamic attention weights using a softmax activation function. These weights reflect the importance of each channel and are multiplied with the original input features to dynamically select and enhance key feature information. The two dynamically selected features are concatenated and fused. 1×1 and 3×3 convolution blocks are applied to the concatenated features to further aggregate and enhance the fused features. The resulting fused cue features contain complementary information from keypoint detection and segmentation tasks, providing more accurate and rich prior information for subsequent detection models.

[0074] S220: Acquire detection output features of the target detection model, and process the fused prompt features and the detection output features based on a prompt information interaction module to obtain target detection features.

[0075] The detection output features may be multi-level output features corresponding to the image processed by the target detection model. The prompt information interaction module may be understood as a neural network layer for integrating features into a fusion process.

[0076] Specifically, the structure of the Prompt interaction module is as follows Figure 7As shown in Figure 1, this module effectively integrates the fused prompt features with the multi-level output features of the YOLOv8 backbone. The Prompt Interaction Module consists of two main components: an attention module and a feed-forward network (FFN). First, features from a certain level of the YOLOv8 backbone are concatenated with the fused prompt features. The concatenated features are then fed into the Attention module, which follows the basic attention mechanism of the Transformer architecture. In the Attention module, the concatenated features undergo preliminary processing via a 3×3 convolution block to enhance their expressiveness. Channel separation and shape transformation are then performed to generate a query vector (Q) and a key vector (K). The concatenated features are then aggregated and reshaped via a 1×1 convolution to produce a value vector (V). After the attention operation, the Q, K, and V vectors fuse the relevant prompt information with the original features. At this point, the fused features are residually connected with the input features to form the final feature representation that embeds the prompt information. Next, the features embedded with the prompt information are fed into a feed-forward network (FFN) consisting of gated transposed convolutions. This module decouples the input features using a cascade of two 1×1 convolution blocks and a 3×3 convolution block. This produces two branch features, one of which is gated attention information after a sigmoid function. This branch is then dot-multiplied with the other branch feature, aggregated through a 1×1 convolution, and finally concatenated with the input feature residual to produce the final output feature. In this way, the Prompt Interaction Module effectively integrates prompt information from different models, providing more semantically rich and spatially expressive features for subsequent detection tasks.

[0077] S230: The target detection model outputs a through-hole defect detection label according to the target detection feature.

[0078] Specifically, the object detection model ultimately outputs a through-hole defect detection label corresponding to the image based on the object detection features. It should be noted that to validate the effectiveness of the pruning and compression method, detailed comparative experiments were conducted and evaluated using a self-constructed test set. First, a baseline model (YOLOV8-S) was trained, and several key performance metrics on the test set were recorded, including detection accuracy, parameter count, and inference time. The baseline model is a complete, unpruned network. After thorough training, its detection performance reached optimal levels. Experimental results show that the baseline model achieved an accuracy of 83.32%, 7.2M parameters, and an inference time of 302.23ms / image. After achieving optimal performance for the baseline model, the pruning and compression method was applied to optimize the model. The pruning operation involves channel pruning and feature reconstruction. Pruning significantly reduces computational complexity and parameter count. Specifically, the pruned model's parameter count was reduced by 73.2%, and inference time was reduced from 302.23ms / image to 63.3ms / image. Although pruning significantly reduces the amount of computation and parameters, the performance of the model decreases to a certain extent due to the loss of some information during the pruning process. In our experiments, the accuracy of the pruned model decreased by about 1.05%.

[0079] In the multi-model fusion process provided by an embodiment of the present invention, the key point detection model and segmentation model are first pruned and trained in the same manner. The pruned key point detection model and segmentation model are used as feature generators, and key point information and segmentation mask information are extracted from each model. The two features are fused through the Prompt fusion module to generate a fused feature containing multimodal information. These fused features are then input into an object detection model (such as YOLOv8) for further processing. In multi-model fusion, the parameters of the pruned pre-trained key point detection model and segmentation model are frozen, preventing them from being updated during training. They only serve as feature generators to provide information. This ensures that the feature information they provide is not interfered with by the training process, ensuring the stability of the multimodal features. When the multi-level output features of the YOLOv8 Backbone are fused with the fused prompt features, the Prompt interaction module further combines the features of the two to generate the final feature representation used for detection. Through multi-model fusion, the detection accuracy of the pruned model is significantly improved. Compared to a single pruned model, the detection accuracy of the multi-model fusion model increased by 13.87%. Furthermore, the fused model not only restored accuracy but also achieved a better balance between inference speed and memory usage. The inference time of the multi-model fusion model was comparable to that of the baseline model, and the inference latency was reduced by 20% compared to the single pruned model.

[0080] The technical solution of the embodiment of the present invention obtains a set of sample images to be processed, processes the set of sample images to be processed to obtain a target sample image set, and prunes and trains each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained. Finally, a target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, meeting real-time requirements.

[0081] Example 3:

[0082] Figure 8 A schematic diagram of the structure of a multi-model fusion PCB through-hole defect detection device provided by an embodiment of the present invention. Figure 8 As shown, the device includes: a sample image determination module 810, a model training module 820 and a defect detection module 830, wherein:

[0083] The sample image determination module 810 is configured to obtain a sample image set to be processed and process the sample image set to be processed to obtain a target sample image set;

[0084] a model training module 820 configured to perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model, and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other;

[0085] The defect detection module 830 is used to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the through-hole defect detection result corresponding to the image to be detected.

[0086] Based on the above technical solution, the model training module is used to determine the scaling factor of each channel corresponding to the batch normalization layer in each deep learning model to be trained; regularize the scaling factor of each channel, and prune the channel of the batch normalization layer according to the regularization processing result.

[0087] Based on the above technical solution, the model training module is used to determine the optimal reconstruction matrix and the optimal model parameters based on minimizing the reconstruction error function; the feature reconstruction layer is deployed after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein the feature reconstruction layer is a structure of three layers of convolution in series.

[0088] Based on the above technical solution, the defect detection module is used to remove the output layer in the target key point model and the target segmentation model, and input the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module; and establish the target fusion model based on the fused prompt features and the target detection model.

[0089] Based on the above technical solution, the defect detection module is used to obtain the detection output features of the target detection model, and process the fused prompt features and the detection output features based on the prompt information interaction module to obtain the target detection features; the target detection model outputs the through-hole defect detection label according to the target detection features.

[0090] On the basis of the above technical solution, the sample image determination module is used to annotate the sample image set to be processed based on image annotation software to obtain a sample image set containing annotated labels; wherein the annotated labels include key point labels, target detection labels and grayscale labels;

[0091] The sample image set is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

[0092] Based on the above technical solution, the sample image determination module is used to adjust the image sizes in the sample image set and the test image set based on the preset image size, and perform random horizontal flipping processing; convert the images subjected to random horizontal flipping processing into a tensor mode format, and perform normalization processing to obtain the target sample image set.

[0093] The technical solution of the embodiment of the present invention obtains a set of sample images to be processed, processes the set of sample images to be processed to obtain a target sample image set, and prunes and trains each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained. Finally, a target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, meeting real-time requirements.

[0094] The multi-model fusion PCB through-hole defect detection device provided in an embodiment of the present invention can execute the multi-model fusion PCB through-hole defect detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Example 4:

[0096] Figure 9 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0097] like Figure 9 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0098] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0099] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the multi-model fusion PCB through-hole defect detection method.

[0100] In some embodiments, the multi-model fusion PCB through-hole defect detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multi-model fusion PCB through-hole defect detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the multi-model fusion PCB through-hole defect detection method by any other appropriate means (for example, by means of firmware).

[0101] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0106] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0107] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0108] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A multi-model fusion PCB through-hole defect detection method, characterized in that: include: Acquire a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; Pruning the deep learning models to be trained based on the target sample image set to obtain target deep learning models corresponding to the deep learning models to be trained, wherein the target deep learning models include a target key point model, a target detection model, and a target segmentation model; the model structures of the deep learning models to be trained are different from each other; and a decoding head using deconvolution is added to the output layer of the target key point detection model; Establishing a target fusion model based on the target deep learning model, inputting the image to be detected into the target fusion model, and determining a through-hole defect detection result corresponding to the image to be detected; The target fusion model is established based on the target deep learning model, including: removing the output layer in the target key point model and the target segmentation model, and inputting the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fusion prompt feature output by the prompt information fusion module; obtaining the detection output feature of the target detection model, and processing the fusion prompt feature and the detection output feature based on the prompt information interaction module to obtain the target detection feature; the target detection model outputs a through-hole defect detection label according to the target detection feature; the prompt information interaction module includes an attention module and a feedforward network; the attention module is used to process the splicing feature to obtain the feature of embedded prompt information; the feedforward network is used to process the feature of embedded prompt information to obtain the target detection feature; the splicing feature is obtained by splicing the detection output feature and the fusion prompt feature.

2. The method according to claim 1, characterized in that The pruning training of each deep learning model to be trained based on the target sample image set includes: Determining a scaling factor for each channel corresponding to a batch normalization layer in each of the deep learning models to be trained; Regularization processing is performed on the scaling factors of the channels, and channel pruning is performed on the batch normalization layer according to the regularization processing results.

3. The method according to claim 2, characterized in that After performing channel pruning on the batch normalization layer according to the regularization processing result, the method further includes: Determine the optimal reconstruction matrix and optimal model parameters based on minimizing the reconstruction error function; A feature reconstruction layer is deployed after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein the feature reconstruction layer is a structure of three layers of convolution in series.

4. The method according to claim 1, wherein The step of processing the sample image set to be processed to obtain a target sample image set includes: Annotating the sample image set to be processed based on image annotation software to obtain a sample image set containing annotated labels; wherein the annotated labels include key point labels, target detection labels, and grayscale labels; The sample image set is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

5. The method according to claim 4, characterized in that The determining the target sample image set based on the sample image set and the test image set includes: Adjust the image sizes of the sample image set and the test image set based on a preset image size, and perform random horizontal flipping processing; The image subjected to the random horizontal flipping process is converted into a tensor format and normalized to obtain the target sample image set.

6. A multi-model fusion PCB through-hole defect detection device, characterized in that: include: A sample image determination module is used to obtain a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; A model training module, configured to perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model, and a target segmentation model; the model structures of each deep learning model to be trained are different from each other; and a decoding head using deconvolution is added to the output layer of the target key point detection model; A defect detection module is configured to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine a through-hole defect detection result corresponding to the image to be detected; The defect detection module is used to remove the output layer in the target key point model and the target segmentation model, and input the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fused prompt feature output by the prompt information fusion module; obtain the detection output feature of the target detection model, and process the fused prompt feature and the detection output feature based on the prompt information interaction module to obtain the target detection feature; the target detection model outputs a through-hole defect detection label according to the target detection feature; the prompt information interaction module includes an attention module and a feedforward network; the attention module is used to process the splicing feature to obtain the feature of embedded prompt information; the feedforward network is used to process the feature of embedded prompt information to obtain the target detection feature; the splicing feature is obtained by splicing the detection output feature and the fused prompt feature.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-model fusion PCB through-hole defect detection method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-model fusion PCB through-hole defect detection method according to any one of claims 1 to 5 when executed.

Images