Multi-model fusion PCB through hole defect detection method, device, equipment and medium

Abstract

The invention discloses a multi-model fusion PCB through hole defect detection method, device and equipment and a medium, and the method comprises the steps: obtaining a to-be-processed sample image set, and processing the to-be-processed sample image set to obtain a target sample image set; performing pruning training on each to-be-trained deep learning model based on the target sample image set to obtain a target deep learning model corresponding to each to-be-trained deep learning model, the target deep learning model including a target key point model, a target detection model and a target segmentation model; the model structures of the to-be-trained deep learning models are different from each other; and establishing a target fusion model based on the target deep learning model, inputting the to-be-detected image into the target fusion model, and determining a through hole defect detection result corresponding to the to-be-detected image. Through multi-model fusion and pruning optimization, the advantages of each model in defect detection can be fully played, and the reasoning speed is remarkably improved on the basis of ensuring high precision.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-model fusion PCB through-hole defect detection method, device, equipment and medium. Background Art

[0002] With the rapid development of the electronics manufacturing industry, the quality of printed circuit boards (PCBs), as the foundation and core components of electronic products, is crucial to the performance and reliability of products. PCB vias are key structures used to connect different layers in circuit boards and are widely used in the manufacture of multi-layer circuit boards. During the PCB production process, vias may have defects due to various reasons, such as uneven apertures, cracks in the hole wall, copper foil shedding, etc. These defects directly affect the electrical performance and reliability of the circuit and may even cause product failure.

[0003] However, traditional through-hole defect detection methods mostly rely on manual visual inspection and traditional optical inspection equipment, which have shortcomings such as low efficiency and being greatly affected by human factors. Summary of the invention

[0004] The present invention provides a multi-model fusion PCB through-hole defect detection method, device, equipment and medium. Through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved on the basis of ensuring high accuracy, meeting real-time requirements, and solving the problem of reduced model reasoning speed in the prior art.

[0005] According to one aspect of the present invention, a multi-model fusion PCB through-hole defect detection method is provided, which includes: Acquire a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; Based on the target sample image set, pruning training is performed on each deep learning model to be trained to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other; A target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected.

[0006] According to another aspect of the present invention, a multi-model fusion PCB through-hole defect detection device is provided, which includes: A sample image determination module is used to obtain a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; A model training module, used for performing pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other; The defect detection module is used to establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the PCB through-hole defect detection method of multi-model fusion described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-model fusion PCB through-hole defect detection method described in any embodiment of the present invention when executed.

[0009] The technical solution of the embodiment of the present invention is to obtain a set of sample images to be processed, process the set of sample images to be processed to obtain a target sample image set, and prune and train each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, and finally establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved on the basis of ensuring high accuracy to meet real-time requirements.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 It is a flow chart of a multi-model fusion PCB through-hole defect detection method provided according to an embodiment of the present invention.

[0013] Figure 2 is a flow chart of a key point detection model provided according to an embodiment of the present invention.

[0014] Figure 3 is a schematic diagram of a channel scaling factor provided according to an embodiment of the present invention.

[0015] Figure 4 It is a flow chart of a multi-model fusion PCB through-hole defect detection method provided according to an embodiment of the present invention.

[0016] Figure 5 It is a schematic diagram of the target fusion model processing process provided according to an embodiment of the present invention.

[0017] Figure 6 It is a schematic diagram of the processing process of the information fusion module provided according to an embodiment of the present invention.

[0018] Figure 7 It is a schematic diagram of the processing process of the prompt information interaction module provided according to an embodiment of the present invention.

[0019] Figure 8 It is a structural schematic diagram of a multi-model fusion PCB through-hole defect detection device provided according to an embodiment of the present invention.

[0020] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] Embodiment 1: Figure 1 A flowchart of a multi-model fusion PCB through-hole defect detection method provided in an embodiment of the present invention is applicable to the case where, during the PCB production process, through-hole defects of a PCB are automatically detected based on the collected PCB image. The method can be executed by a multi-model fusion PCB through-hole defect detection device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes: S110: Acquire a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set.

[0024] The sample image set to be processed may be an unprocessed original image obtained after scanning the PCB, and the target sample image set may be understood as a set of sample images obtained after processing the original image.

[0025] Specifically, the PCB is scanned by an X-ray device to obtain an unprocessed PCB image, and the sample image set to be processed is processed to obtain a target sample image set. For example, the images in the sample image set to be processed may be cleaned and enhanced to obtain a target sample image set.

[0026] On the basis of the above technical solution, the processing of the sample image set to be processed to obtain a target sample image set includes: annotating the sample image set to be processed based on image annotation software to obtain a sample image set containing annotated labels; dividing the sample image set into a sample image set and a test image set according to a preset ratio, and determining the target sample image set based on the sample image set and the test image set.

[0027] Among them, the image annotation software can be software for annotating images to be annotated, and can be labelme software. The annotation labels can be pre-set image labels, and the annotation labels include key point labels, target detection labels, and grayscale labels. The preset ratio can be understood as a pre-set ratio for dividing the sample set. The sample image set can be an image set for training the model, and correspondingly, the test image set can be an image set for testing the model.

[0028] Specifically, appropriate image annotation software is selected according to needs, such as LabelImg, LabelMe, VOTT and other annotation software. The annotated labels include three categories: key points, target detection and mask. Then, the rotating annotation software is used to annotate the images in the sample image set to be processed, and after annotation, the sample image set is divided into a sample image set and a test image set according to a preset ratio. For example, it can be divided into a training set and a test set in a ratio of 7:3.

[0029] On the basis of the above technical solution, the target sample image set is determined based on the sample image set and the test image set, including: adjusting the image sizes in the sample image set and the test image set based on a preset image size, and performing random horizontal flipping processing; converting the images subjected to random horizontal flipping processing into a tensor format, and performing normalization processing to obtain the target sample image set.

[0030] The preset image size can be understood as the size of the input image that is preset. The tensor format can be a format for describing the image as a tensor.

[0031] Specifically, the data is preprocessed, the input image size is changed to 1024×1024, and then randomly flipped horizontally with a probability of 0.5. After the image is converted to Tensor format, it is standardized and normalized. For example, with the help of the Python language environment, the torchvision.Resize() method is used to adjust the image size. Subsequently, the torchvision.RandomCrop() method is applied to randomly crop the image, and then the torchvision.RandomHorizontalFlip() method is used to perform a horizontal random flip operation on the image with a probability of 50%. Next, the processed image is converted to Tensor format through the transforms.ToTensor() method, and preliminary normalization processing is implemented in the process. Finally, the transforms.Normalize() function is used to complete the standardization processing step of the image data to ensure that the data meets the predefined normalization and standardization requirements. The specific algorithm details of the standardization and normalization include: ; where μ represents the mean of the image, x represents the tensor of the image, σ represents the standard deviation, and x max Indicates the maximum value of the image tensor, x min Represents the minimum value of the image tensor, Represents the normalized image tensor, Represents the normalized image tensor.

[0032] S120. Perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained.

[0033] The deep learning model to be trained may be an initialized deep learning model, which may include a key point model to be trained, a detection model to be trained, and a segmentation model to be trained. The target deep learning model includes a target key point model, a target detection model, and a target segmentation model; the model structures of the deep learning models to be trained are different from each other.

[0034] Specifically, we build key point models, detection models, and segmentation models, using LiteHRnet, YOLOV8, and Unet models respectively, and further initialize the key point models, detection models, and segmentation models, design a structured pruning training framework, and perform pruning training on the key point models, detection models, and segmentation models respectively. Figure 2 As shown in the figure, since the size of the through hole accounts for a very small proportion of the entire image, an additional decoding head is added to the output layer of the LiteHRnet model, and an additional resolution map is output through the decoding head. Deconvolution is used in the additional decoding head, and finally a HeatMap of the original size is output. Since the key point detection model is only used to locate the target position, it can be understood as a multi-target single-type key point detection model, so the detection logic of the bottom-up framework is adopted. Referring to the CenterNet model, the output of LiteHRnet is a Heatmap feature map, and the training labels are the Heatmap feature map and mask. The mask is generated according to the coordinates corresponding to the maximum value of each Gaussian kernel in the Heatmap label. The FocalLoss loss function is used to calculate the Heatmap feature loss. Assume that the input image is , the output Heatmap is , FocalLoss can be expressed as: ;in, and are hyperparameters and are set to 2 and 4 respectively.

[0035] YOLOv8 uses the lightest version YOLOV8-s, whose labels consist of three parts: bounding box parameters, category labels, and confidence. Based on gridcell, each grid predicts multiple bounding boxes. The label of each bounding box includes: the center point of the bounding box, the width and height of the bounding box, and the probability distribution of the confidence and category. Its form is The loss function of YOLOv8 consists of multiple parts, including:LocalizationLoss: used to measure the difference between the predicted bounding box and the true bounding box. Usually IoU (Intersection over Union) and MSE (mean square error) are used to calculate: ; Confidence Loss is used to measure the difference between predicted confidence and actual confidence. Binary crossentropy (BCE) is often used to calculate: ; ClassificationLoss is used to measure the difference between the predicted category and the true category, usually calculated using cross-entropy loss: ; Then we can determine the final loss function combination as: ; The encoder of the segmentation model Unet uses ResNet50. The label is to mark the region of the through-hole position. The region of the through-hole position is extracted by binary classification. The loss function is: .

[0036] On the basis of the above technical solution, the pruning training of each deep learning model to be trained is performed based on the target sample image set, including: determining the scaling factor of each channel corresponding to the batch normalization layer in each deep learning model to be trained; regularizing the scaling factor of each channel, and pruning the channel of the batch normalization layer according to the regularization processing result.

[0037] Among them, the batch normalization layer can be used to standardize the input of each layer of the neural network, which can solve the distribution drift of input data during training.

[0038] Specifically, in the deep learning model, find all batch normalization layers, and for each batch normalization layer, extract its corresponding scaling factor. It should be noted that the scaling factor determines the importance of each channel in the feature map. Then, during the model training process, the regularization term is added to the loss function. The pruning threshold is determined based on the scaling factor after regularization, and channels below the pruning threshold will be pruned. For example, in deep learning, the BatchNormalization (BN) layer is a commonly used technique to accelerate training and improve the generalization ability of the model. The core operation of the batch normalization layer is to standardize the input of each layer to ensure that its mean is 0 and the variance is 1. The batch normalization layer usually includes two important learnable parameters: the scaling factor (scalefactor) and offset factor .

[0039] In order to achieve batch normalization layer pruning, these parameters are regularized to reduce unimportant channels and weights. Batch normalization layer pruning mainly selects the channels that need to be pruned by evaluating the importance of each channel. Through L1 regularization, the model can automatically reduce the importance of unimportant channels. The parameters are compressed to zero, thereby achieving channel pruning. For an input tensor , the operation process of the batch normalization layer can be expressed by the following formula: ;in, Yes Input The mean value in the batch dimension, is the variance of the input, is a small constant used to prevent the denominator from being zero. The batch normalization layer then scales and offsets the normalized input: ;in, is the scaling factor for each channel, is the offset factor for each channel.

[0040] The goal of batch normalization layer pruning is to regularize the scaling factor , remove those channels that contribute less to the model. The specific approach is to evaluate the importance of each channel through L1 regularization and The value of determines which channels can be pruned. Channels with less importance will be pruned, thereby reducing the amount of computation and the number of parameters of the network. Figure 3 For each channel in the batch normalization layer , whose scaling factor Its importance can be assessed by adding L1 regularization. The goal of L1 regularization is to make The weights of channels with smaller values ​​tend to zero. The L1 regularization term can be expressed as: ; The L1 regularization term is applied to the loss function through the gradient descent optimization process, which promotes some unimportant is reduced or becomes zero. The total loss function becomes: .

[0041] On the basis of the above technical solution, after performing channel pruning on the batch normalization layer according to the regularization processing result, it also includes: determining the optimal reconstruction matrix and the optimal model parameters based on minimizing the reconstruction error function; and deploying the feature reconstruction layer after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters.

[0042] Among them, the minimization of the reconstruction error function can be a loss function for determining the feature reconstruction layer. The reconstruction matrix is ​​a matrix used to convert the pruned feature map back to a form close to the original feature map, thereby reducing the information loss caused by pruning. The feature reconstruction layer is a structure of three layers of convolution in series.

[0043] Specifically, a reconstruction error function is constructed. The reconstruction error function can measure the difference between the output of the pruned model and the output of the original model. Then, the reconstruction matrix is ​​solved through an optimization algorithm (such as gradient descent, least squares method, etc.) to minimize the reconstruction error function. While solving the reconstruction matrix, the parameters of the model also need to be updated to ensure that the model can still maintain good performance after pruning and reconstruction. Then, a feature reconstruction layer is designed according to the optimal reconstruction matrix. This layer will be applied after the batch normalization layer after pruning. It should be noted that in the process of channel pruning, since some information is removed, it will affect the accuracy of the model. Therefore, the feature map reconstruction step aims to restore the lost feature information to ensure the performance of the model. For some information lost after pruning, a reconstruction network is designed to fit the feature maps before and after pruning to restore the lost information. Assume that the original feature map of the output of the network before pruning is , after pruning, the feature map obtained , , some channels are pruned. Among them, b represents the batch dimension (batch), which is used to indicate the number of samples input at a time. During the training process, inputting multiple samples at a time can improve the computational efficiency. c: represents the channel dimension (channel), which is used to indicate the number of channels of the input data or the number of feature maps. For convolutional neural networks (CNN), it usually represents the number of output feature maps of the convolutional layer. h: represents the height dimension (height), which is used to indicate the height of the image or feature map. w: represents the width dimension width), which is used to indicate the width of the image or feature map.

[0044] We hope to Reconstruct to get a feature map as close to the original as possible The feature map after pruning Input to the reconstruction network Ref. Usually, it is expected that the pruned features have similar output features to the original model after the reconstruction model: ; in, represents the error term, which represents the part that cannot be completely restored by linear combination. The reconstruction module transforms the pruned feature map structure to be the same as the original feature map structure. It should be noted that the feature reconstruction layer is a three-layer convolution structure, using a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution structure. The spatial structure of The sum of squares of , that is, minimizing the reconstruction error: ; among them, among them Represents the Frobenius norm, which is the square root of the sum of the squares of the elements of the matrix. Represents the network parameters before pruning. By minimizing the above loss function, the optimal reconstruction matrix is ​​obtained. and model parameters After completing the training of the reconstructed network, it is usually necessary to fine-tune the pruned network to ensure that the fusion of the reconstructed network and the main network does not affect the model performance.

[0045] S130, establishing a target fusion model based on the target deep learning model, and inputting the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected.

[0046] The target fusion model can be understood as a deep learning model obtained by combining the target deep learning models. The image to be inspected can be a PCB image that needs to be inspected for through-hole defects.

[0047] Specifically, after the target deep learning model training is completed, a target fusion model can be constructed according to each target deep learning model, and the image to be detected is input into the target fusion model to obtain the through-hole defect detection result output by the target fusion model.

[0048] The technical solution of the embodiment of the present invention is to obtain a set of sample images to be processed, process the set of sample images to be processed to obtain a target sample image set, and prune and train each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, and finally establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved on the basis of ensuring high accuracy to meet real-time requirements.

[0049] Embodiment 2: Figure 4 The present invention provides a flowchart of a multi-model fusion PCB through-hole defect detection method. This embodiment further refines the technical solution of establishing a target fusion model based on the target deep learning model on the basis of the above technical solution. Figure 4 As shown, the method includes: S210, removing the output layer in the target key point model and the target segmentation model, and inputting the hidden layer results in the target key point model and the target segmentation model into a prompt information fusion module to obtain a fused prompt feature output by the prompt information fusion module.

[0050] The hidden layer result may be the output result of the last hidden layer in the key point model and the segmentation model. The information fusion module may be understood as a processing model for fusing the output features of the target key point model and the target segmentation model.

[0051] Specifically, the entire fusion structure of the fusion model is as follows Figure 5 As shown in the figure, the key point detection model and the segmentation model load the pruned pre-trained parameters, remove the output layer, and use the last hidden layer features as the output, which are input into the Prompt fusion module to generate a multi-modal prompt prior. During the training process, the key point detection model and the segmentation model will freeze the parameters and will not participate in the network parameter update. As a multi-modal feature generator, the key point information and the mask information are fused through the Prompt fusion module. As a multi-modal prompt feature, it is fused with the multi-level output features of the YOLOV8 Backbone through the Prompt interaction module. The design goal of the Prompt interaction module is to establish an effective connection between different modal features so that the key point information and the mask information can provide more accurate context information for the YOLOv8 detection model, thereby enhancing the recognition ability of the model in through-hole defect detection.

[0052] It should be noted that the structure of the Prompt fusion module is as follows Figure 6 As shown in the figure. Since the output features of the segmentation model and the key point model are not aligned in the spatial dimension and channel dimension, a 1×1 convolution is first used to transform the channels of the two features to ensure that they have the same number of channels. Then, the features are enhanced through a 3×3 convolution block to extract richer feature information and improve the representation ability of the features. The enhanced features adjust their spatial resolution through spatial transformation so that the spatial scales of the two features are aligned. However, spatial transformation may introduce some interference and unnecessary noise, so after that, another 3×3 convolution block is used to suppress the interference caused by spatial transformation to ensure feature quality and effectiveness. In order to avoid the suboptimal effect that may be caused by static fusion, a channel attention mechanism is introduced to dynamically adjust the importance of fused features. Before feature concatenation, the input features first obtain global information through a global average pooling operation, then undergo channel dimension reduction through a 1×1 convolution, and then generate dynamic attention weights through a Softmax activation function. These weights reflect the importance of each channel and are multiplied by the original input features to dynamically select and enhance key feature information. The two features after dynamic selection will be spliced ​​and fused, and 1×1 convolution and 3×3 convolution blocks will be applied to the spliced ​​features to further aggregate and enhance the fused features. The final generated fusion hint feature contains complementary information from key point detection and segmentation tasks, providing more accurate and rich prior information for subsequent detection models.

[0053] S220, obtaining the detection output feature of the target detection model, and processing the fused prompt feature and the detection output feature based on the prompt information interaction module to obtain the target detection feature.

[0054] The detection output feature may be a multi-level output feature corresponding to the image processed by the target detection model. The prompt information interaction module may be understood as a neural network layer for integrating the features into a fusion process.

[0055] Specifically, the structure of the Prompt interaction module is as follows Figure 7As shown in Figure 1, this module aims to effectively fuse the fused prompt features with the multi-level output features of the YOLOv8 Backbone. The Prompt Interaction Module consists of two main parts: the Attention module and the Feedforward Network (FFN). First, a certain level of features in the YOLOv8 Backbone is concatenated with the fused prompt features. The concatenated features are input into the Attention module, which follows the basic attention mechanism in the Transformer structure. In the Attention module, the concatenated features are first processed by a 3×3 convolution block to enhance the expressiveness of the features. Then, channel separation and shape transformation operations are performed to generate the query vector (Q) and key vector (K). Then, the concatenated features are aggregated and reshaped by 1×1 convolution to obtain the value vector (V). After the attention operation, the Q, K, and V vectors will fuse the relevant prompt information with the original features. At this point, the fused features will be residually connected with the input features to form the final feature representation containing the embedded prompt information. Next, the features embedded with the prompt information will be passed into a feed-forward network (FFN) consisting of gated transposed convolutions. This module decouples the input features using a block cascaded with two 1×1 convolutions and a 3×3 convolution block to obtain two branch features, one of which is gated attention information after a Sigmoid function, which is point-multiplied with the other branch feature and aggregated through a 1×1 convolution, and finally linked with the input feature residual to obtain the final output feature. In this way, the Prompt interaction module can effectively fuse prompt information from different models, providing more semantically rich and spatially expressive features for subsequent detection tasks.

[0056] S230. The target detection model outputs a through-hole defect detection label according to the target detection feature.

[0057] Specifically, the target detection model finally outputs the through-hole defect detection label corresponding to the image according to the target detection feature. It should be noted that in order to verify the effect of the pruning compression method, a detailed comparative experiment was carried out and evaluated using a self-built test set. First, the benchmark model (YOLOV8-S) was trained and its multiple key performance indicators on the test set were recorded, including detection accuracy, parameter amount, and inference time. The benchmark model is a complete network without pruning. After sufficient training, it is ensured that its detection performance has reached the best level. According to the experimental results, the accuracy of the benchmark model is 83.32%, the parameter amount is 7.2 M, and the inference time is 302.23ms / image. After obtaining the best performance of the benchmark model, the pruning compression method was applied to optimize the model. The pruning operation includes two parts: channel pruning and feature reconstruction. After pruning, the computational complexity and parameter amount are significantly reduced, which is specifically manifested in that the parameter amount of the pruned model is reduced by 73.2%, and the inference time is reduced from 302.23ms / image to 63.3ms / image. Although pruning significantly reduces the amount of computation and parameters, the performance of the model has declined to a certain extent due to the loss of some information during the pruning process. In our experiments, the accuracy of the pruned model decreased by about 1.05%.

[0058] In the multi-model fusion process provided by the embodiment of the present invention, the key point detection model and the segmentation model are first pruned and trained in the same way, and the pruned key point detection model and segmentation model are used as feature generators to extract key point information and segmentation mask information from each model respectively. After the two features are fused by the Prompt fusion module, a fusion feature containing multimodal information is generated. These fused features are input into the target detection model (such as YOLOv8) for further processing. In multi-model fusion, the parameters of the key point detection model and segmentation model that have been pruned and pre-trained are frozen to avoid updating them during the training process, and only provide information as a feature generator. In this way, it can be ensured that the feature information they provide will not be disturbed by the training process, ensuring the stability of the multimodal features. When the multi-level output features of the YOLOv8 Backbone are fused with the fused prompt features, the features of the two are further combined through the Prompt interaction module to generate the final feature representation for detection. Through the multi-model fusion method, the detection accuracy of the pruned model has been significantly improved. Compared with a single pruned model, the detection accuracy after multi-model fusion increased by 13.87%. In addition, the fused model not only restored the accuracy, but also achieved a better balance in inference speed and memory usage. The inference time of the multi-model fusion model was not much different from that of the baseline model, and the inference delay was reduced by 20% compared with the single pruned model.

[0059] The technical solution of the embodiment of the present invention is to obtain a set of sample images to be processed, process the set of sample images to be processed to obtain a target sample image set, and prune and train each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, and finally establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved on the basis of ensuring high accuracy to meet real-time requirements.

[0060] Embodiment three: Figure 8 A schematic diagram of the structure of a multi-model fusion PCB through-hole defect detection device provided by an embodiment of the present invention. Figure 8 As shown, the device includes: a sample image determination module 810, a model training module 820 and a defect detection module 830, wherein: The sample image determination module 810 is used to obtain a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; A model training module 820 is used to perform pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other; The defect detection module 830 is used to establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected.

[0061] Based on the above technical solution, the model training module is used to determine the scaling factor of each channel corresponding to the batch normalization layer in each deep learning model to be trained; regularize the scaling factor of each channel, and prune the channel of the batch normalization layer according to the regularization result.

[0062] Based on the above technical solution, the model training module is used to determine the optimal reconstruction matrix and the optimal model parameters based on minimizing the reconstruction error function; the feature reconstruction layer is deployed after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein the feature reconstruction layer is a structure of three layers of convolution in series.

[0063] On the basis of the above technical solution, the defect detection module is used to remove the output layer in the target key point model and the target segmentation model, and input the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module; and establish the target fusion model based on the fused prompt features and the target detection model.

[0064] On the basis of the above technical solution, the defect detection module is used to obtain the detection output features of the target detection model, and to process the fused prompt features and the detection output features based on the prompt information interaction module to obtain the target detection features; the target detection model outputs a through-hole defect detection label according to the target detection features.

[0065] On the basis of the above technical solution, the sample image determination module is used to annotate the sample image set to be processed based on image annotation software to obtain a sample image set containing annotated labels; wherein the annotated labels include key point labels, target detection labels and grayscale labels; The sample image set is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

[0066] On the basis of the above technical solution, the sample image determination module is used to adjust the image size in the sample image set and the test image set based on a preset image size, and perform random horizontal flipping processing; convert the image subjected to random horizontal flipping processing into a tensor mode format, and perform standardization processing to obtain the target sample image set.

[0067] The technical solution of the embodiment of the present invention is to obtain a set of sample images to be processed, process the set of sample images to be processed to obtain a target sample image set, and prune and train each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, and finally establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through-hole defect detection result corresponding to the image to be detected. Based on the above technical solution, through multi-model fusion and combined with pruning optimization, the advantages of each model in defect detection can be fully utilized, and the reasoning speed can be significantly improved on the basis of ensuring high accuracy to meet real-time requirements.

[0068] The multi-model fusion PCB through-hole defect detection device provided in the embodiment of the present invention can execute the multi-model fusion PCB through-hole defect detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0069] Embodiment 4: Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0070] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0071] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0072] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a multi-model fusion PCB through-hole defect detection method.

[0073] In some embodiments, the multi-model fusion PCB through-hole defect detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multi-model fusion PCB through-hole defect detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the multi-model fusion PCB through-hole defect detection method by any other appropriate means (e.g., by means of firmware).

[0074] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0076] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0078] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0079] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0080] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0081] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-model fusion PCB through-hole defect detection method, characterized in that: include: Acquire a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; Based on the target sample image set, pruning training is performed on each deep learning model to be trained to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other; Establishing a target fusion model based on the target deep learning model, and inputting the image to be detected into the target fusion model to determine a through-hole defect detection result corresponding to the image to be detected; The target fusion model is established based on the target deep learning model, including: removing the output layer in the target key point model and the target segmentation model, and inputting the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fusion prompt feature output by the prompt information fusion module; acquiring the detection output feature of the target detection model, and processing the fusion prompt feature and the detection output feature based on the prompt information interaction module to obtain the target detection feature; the target detection model outputs a through-hole defect detection label according to the target detection feature.

2. The method according to claim 1, characterized in that The pruning training of each deep learning model to be trained based on the target sample image set includes: Determine a scaling factor for each channel corresponding to a batch normalization layer in each of the deep learning models to be trained; Regularization processing is performed on the scaling factors of the channels, and channel pruning is performed on the batch normalization layer according to the regularization processing result.

3. The method according to claim 2, characterized in that After performing channel pruning on the batch normalization layer according to the regularization processing result, the method further includes: Determine the optimal reconstruction matrix and optimal model parameters based on minimizing the reconstruction error function; The feature reconstruction layer is deployed after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein the feature reconstruction layer is a structure of three layers of convolution in series.

4. The method according to claim 1, characterized in that: The step of processing the sample image set to be processed to obtain a target sample image set includes: Annotating the sample image set to be processed based on image annotation software to obtain a sample image set containing annotation labels; wherein the annotation labels include key point labels, target detection labels and grayscale labels; The sample image set is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

5. The method according to claim 4, characterized in that The determining the target sample image set based on the sample image set and the test image set comprises: Adjust the image sizes in the sample image set and the test image set based on a preset image size, and perform random horizontal flipping processing; The image subjected to random horizontal flipping processing is converted into a tensor format, and is standardized to obtain the target sample image set.

6. A multi-model fusion PCB through-hole defect detection device, characterized in that: include: A sample image determination module is used to obtain a sample image set to be processed, and process the sample image set to be processed to obtain a target sample image set; A model training module, used for performing pruning training on each deep learning model to be trained based on the target sample image set to obtain a target deep learning model corresponding to each deep learning model to be trained, wherein the target deep learning model includes a target key point model, a target detection model and a target segmentation model; and the model structures of each deep learning model to be trained are different from each other; A defect detection module, used to establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine a through-hole defect detection result corresponding to the image to be detected; The defect detection module is used to remove the output layer in the target key point model and the target segmentation model, and input the hidden layer results in the target key point model and the target segmentation model into the prompt information fusion module to obtain the fused prompt feature output by the prompt information fusion module; obtain the detection output feature of the target detection model, and process the fused prompt feature and the detection output feature based on the prompt information interaction module to obtain the target detection feature; the target detection model outputs a through-hole defect detection label according to the target detection feature.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-model fusion PCB through-hole defect detection method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-model fusion PCB through-hole defect detection method described in any one of claims 1 to 5 when executed.

Images