Detection method, model training method, equipment, storage medium and program product

Abstract

The present application provides a defect detection method, a defect detection model training method and apparatus, a storage medium, and a computer program product. The defect detection method includes: performing feature extraction processing on a PCB image to be detected to obtain a feature map to be detected of the PCB image to be detected; performing background modeling processing based on the feature map to be detected to obtain a background mask corresponding to the feature map to be detected; removing background features in the feature map to be detected based on the background mask to obtain a defect feature map; and performing classification processing based on the defect feature map to determine the defect category of the PCB to be detected.

Description

Technical Field

[0001] The present disclosure relates to the field of defect detection technology, and in particular to a detection method, a model training method, a device, a storage medium, and a program product. Background Art

[0002] Printed circuit boards (PCBs) are used in the vast majority of electronic products. With the rapid development of the electronics, communications, and computer industries, PCB products are also developing towards high performance, high complexity, and high density. An efficient and highly accurate PCB defect detection system has become an urgent need in the PCB industry. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a defect detection method, a model training method, a device, a storage medium, and a program product.

[0004] According to a first aspect of the present disclosure, a defect detection method is provided, comprising:

[0005] Performing feature extraction processing on the PCB image to be detected to obtain a feature map to be detected of the PCB image to be detected;

[0006] Performing background modeling processing based on the feature map to be detected to obtain a background mask corresponding to the feature map to be detected;

[0007] Removing background features in the feature map to be detected based on the background mask to obtain a defect feature map;

[0008] Classification processing is performed based on the defect feature map to determine the defect category of the PCB to be inspected.

[0009] According to a second aspect of the present disclosure, a defect detection model training method is provided, comprising performing a scale transformation process on an original scale defect image to obtain a scale-transformed defect image;

[0010] The defect categories of the original scale defect image and the transformed scale image are respectively labeled to obtain the original sample image and the transformed sample image;

[0011] obtaining a first background mask based on the original sample image, and obtaining a second background mask based on the transformed sample image;

[0012] A loss function of the defect detection model is determined based on the first background mask and the second background mask.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to: implement the defect detection method as described in any one of the first aspects of the present disclosure, or implement the defect detection model training method as described in any one of the second aspects of the present disclosure.

[0014] According to the fourth aspect of the present disclosure, a computer-readable storage medium is provided, storing executable instructions, which, when executed by a processor, implement the defect detection method as described in any one of the first aspects of the present disclosure, or implement the defect detection model training method as described in any one of the second aspects of the present disclosure.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, characterized in that when the computer program is executed by a processor, it implements the PCB defect detection method as described in any one of the first aspects of the present disclosure, or implements the defect detection model training method as described in any one of the second aspects of the present disclosure.

[0016] The disclosed embodiments provide a defect detection method for PCB defects. This method utilizes background modeling to generate a background mask. Based on this background mask, the method removes background features from a feature map to be detected, ensuring that the retained features are relevant to determining the PCB defect category. This reduces the interference of background features on the determination of PCB defect categories and improves the accuracy of PCB defect category detection and classification in complex image backgrounds. Furthermore, the defect detection model has a small number of parameters, and the defect feature map obtained after background feature removal contains less feature information than the feature map to be detected. This reduces the computational complexity of subsequent feature learning and classification of the defect feature map, resulting in fast inference speed and meeting the real-time requirements of industrial deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a module of an automatic optical inspection device provided by an embodiment of the present disclosure;

[0018] Figure 2 A schematic diagram of a defect detection method according to an embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of a defect detection model provided by an embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of a feature extraction module provided in an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of a filtering module provided in an embodiment of the present disclosure;

[0022] Figure 6 A flowchart of a method for training a defect detection model provided in an embodiment of the present disclosure;

[0023] Figures 7a to 7f A schematic diagram of multiple original defect images provided by an embodiment of the present disclosure;

[0024] Figures 8a to 8f A schematic diagram of multiple pre-processed defect images provided by an embodiment of the present disclosure;

[0025] Figure 9 A schematic diagram of another defect detection model provided by an embodiment of the present disclosure;

[0026] Figure 10 A schematic diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0031] In related technologies, PCB defects are detected through manual inspection methods. However, manual inspection is affected by subjective factors such as workers' experience, emotions, and physiology, resulting in low detection efficiency and high error rates. Mechanical inspection usually uses contact inspection, which requires physical position adjustment to achieve the best detection effect, and is relatively inefficient. Automated Optical Inspection (AOI), based on machine vision, combines electronics, digital image processing, photoelectric detection, and computer technology, and has developed rapidly in the field of precision industrial device inspection. Compared with traditional manual inspection and mechanical inspection, the AOI method has high detection accuracy, fast speed, low cost, and will not cause any impact on the PCB, making it the mainstream method of PCB quality inspection.

[0032] Figure 1 A schematic diagram of a module of an automatic optical inspection device provided by an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the automated optical inspection equipment includes an image acquisition system 10, a control system 20, and a computer 30. The image acquisition system 10 includes an image acquisition device and a light source. The image acquisition device includes various types of CCD / CMOS cameras. During inspection, light from the light source illuminates the surface of the PCB to be inspected. The image acquisition device then takes a picture of the surface to obtain an image of the PCB to be inspected. The control system 20 is connected to the image acquisition system 10 and is used to control the movement of the image acquisition system 10 and also to support and transport the PCB to be inspected. Computer 30 stores computer programs developed based on machine vision, digital graphics processing, and other technologies, which are used to process the image of the PCB to be inspected to achieve inspection.

[0033] Traditional machine learning and statistical methods, such as threshold segmentation, template matching, and PCA (Principal Component Analysis), have been applied to precision manufacturing fields such as metals and optical devices. However, due to the complex background texture of PCBs and the wide variety and shape of defects, it is difficult to develop computer programs suitable for PCB defect detection using these technologies.

[0034] Deep learning models have achieved promising results in feature extraction, classification, and detection tasks, and hold promise for application in PCB inspection. However, due to industry factors, publicly available PCB datasets are very limited, and accurate industrial data annotation requires significant labor costs. Furthermore, PCBs inherently have complex background textures, rich details, and relatively small defect targets, making them prone to false and missed defect detections.

[0035] Based on one or more of the above problems, the embodiments of the present disclosure provide a PCB defect detection method, which can accurately classify PCB defects under non-high-precision labeling and improve industrial production efficiency. Figure 2 A flow chart of a defect detection method provided by an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the defect detection method includes:

[0036] S110: performing feature extraction processing on the PCB image to be inspected to obtain a feature map to be inspected corresponding to the PCB image to be inspected;

[0037] S120: performing background modeling processing based on the feature map to be detected to obtain a background mask corresponding to the feature map to be detected;

[0038] S130: removing background features in the feature map to be detected based on the background mask to obtain a defect feature map;

[0039] S140: Perform classification processing based on the defect feature map to determine the defect category of the PCB to be inspected.

[0040] In an embodiment of the present disclosure, the image of the PCB to be inspected includes an image of a bare PCB board, which refers to a PCB to which electronic components have not yet been soldered. The surface of the PCB includes solder pads and metal traces covered by solder resist. For example, the solder pads can be copper pads, gold pads, etc., and solder paste can also be pre-applied on the surface of the pads. The color of the solder resist can be green, blue, or black, etc. The present disclosure does not limit the material and color of the solder pads on the PCB surface and the color of the solder resist. The present disclosure also does not limit the size of the image of the PCB to be inspected.

[0041] In some embodiments, before step S110, the inspection method further includes: acquiring an image of the PCB to be inspected. The image of the PCB to be inspected is acquired using an image acquisition device, which includes but is not limited to a camera or a webcam. For example, the image acquisition device includes a CCD camera and / or a CMOS camera in an automated optical inspection device.

[0042] The PCB image to be inspected can be a raw image directly captured by the image acquisition device, or it can be an image obtained by preprocessing the raw image. To improve the quality of the PCB image to be inspected and the inspection efficiency, the raw image can be preprocessed by adjusting brightness and contrast.

[0043] For example, the image of the PCB to be inspected may be a color image or a grayscale image.

[0044] Figure 3 A schematic diagram of a defect detection model provided by an embodiment of the present disclosure. The defect detection model is used to implement Figure 2 The defect detection method shown is used to determine the defect category of the PCB to be inspected. Figure 3As shown, the defect detection model may include: a feature extraction module 110, a filtering module 120 and a classification module 130. The feature extraction module 110 is mainly used for low-level feature learning.

[0045] In some embodiments, step S110 is implemented by using the feature extraction module 110 to perform multiple levels of downsampling and upsampling operations on the PCB image to be inspected, obtaining a feature map corresponding to each level. The feature map of each level except the last level is used as input for the downsampling or upsampling process of the next level, and the last level outputs the feature map to be inspected. For example, the feature map to be inspected and the PCB image to be inspected have the same size.

[0046] For example, Figure 4 As shown, the feature extraction module 110 includes a convolution block using a fully convolutional network. The fully convolutional network can include multiple convolutional layers 111 and deconvolution layers 112, but does not include a fully connected layer. By using the fully convolutional network to perform feature extraction processing on the PCB image to be inspected, a feature map to be inspected can be obtained. Specifically, the convolutional layer 111 performs a convolution operation on the input image to extract feature information from the image. The convolution operation reduces the size of the feature map, so a deconvolution layer 112 is required after the last convolutional layer. The deconvolution layer 112 is used to upsample the feature map output by the last convolutional layer to restore the feature map to the same size as the PCB image to be inspected. For example, the fully convolutional network can also include a pooling layer. The pooling layer can be set between adjacent convolutional layers to downsample the feature map output by the convolutional layer to extract key features and reduce the size of the feature map.

[0047] An image can be represented as a three-dimensional pixel matrix with dimensions H × W × D (height × width × depth). Height and width represent the image size, and depth represents the number of channels. A grayscale image has one channel, while a color image has three channels: red, green, and blue, and a depth of three. For example, a color image of a PCB to be inspected can be represented as 224 × 224 × 3. Each value in the matrix represents the value of a pixel.

[0048] The feature map output by each level of the fully convolutional network is expressed in the form of a matrix as H×W×D (height×width×depth), where the height and width represent the size of each feature map, and the depth represents the number of feature maps, which can also be called the number of channels. For example, the feature map to be detected output by the last level of the feature extraction module 110 can be expressed as 224×224×256, which means that the size of each feature map to be detected is 224×224, and the feature map depth is 256. The feature map depth can be set as needed, for example, it can also be 64, 128, 512, 1024, etc. The size of the feature map to be detected and the PCB image to be detected is equal, which means that the width and height values ​​of their corresponding matrices are correspondingly equal. For example, the size of the PCB image to be detected is 224×224, and the size of the feature map to be detected is also 224×224. The present disclosure does not limit the number of feature maps to be detected output by the feature extraction module 110, and the number of feature maps to be detected can be one or more.

[0049] In addition, the present disclosure does not limit the size of the convolution kernel in the convolution layer, the sliding window step size, and the depth of the feature map output at each level in the full convolution network. In a specific embodiment, they can be set as needed. For example, the size of the convolution kernel can be set to 3×3, the sliding window step size can be 1, and the feature map depth can be 32, 64, 128, 256, etc. The present disclosure also does not limit the number of convolution layers and deconvolution layers in the full convolution network. In a specific embodiment, they can be set as needed. For example, the number of convolution layers can be 2 layers, 4 layers, 6 layers, etc., and the number of deconvolution layers can be 2 layers, 4 layers, 6 layers, etc.

[0050] In one embodiment, Figure 4 As shown, the convolution block using the full convolutional network in the feature extraction module 110 can use the UNet structure (encoder-decoder structure), and can also retain low-level detail features through residual connections.

[0051] Continue to see Figure 3 After the feature extraction module 110, a filtering module 120 is introduced to perform background modeling to obtain a background mask 210. The size of the background mask 210 is equal to the size of the feature map to be detected. The background mask 210 is essentially a feature map with a channel number of 1 (i.e., a depth of 1). Each value in the background mask 210 is in the interval [0, 1], and each value represents the probability that the pixel is a defect. The closer the value is to 1, the greater the probability that the pixel is a defect, and the closer the value is to 0, the lower the probability that the pixel is a defect.

[0052] In some embodiments, the filtering module 120 includes a feature fusion layer and an activation layer. Step S120 performs background modeling based on the feature map to be detected to obtain a background mask, specifically including:

[0053] Input the feature map to be detected into the feature fusion layer for feature fusion processing to obtain a fused feature map; wherein the size of the fused feature map and the feature map to be detected are equal;

[0054] The fused feature map is input into the activation layer for normalization to obtain a background mask 210.

[0055] For example, the following combination Figure 5 The process of obtaining the background mask 210 is explained. Figure 3 The filtering module 120 includes a feature fusion layer 121 and an activation layer 122. The feature fusion layer 121 is used to reduce the number of channels of the feature map to be detected to obtain a fused feature map with a single channel, that is, the number of fused feature maps is 1.

[0056] For example, Figure 5 As shown in the figure, the feature fusion layer includes a 1×1 convolution layer. The convolution kernel size of the 1×1 convolution layer is 1×1. Using the 1×1 convolution kernel to perform a convolution operation on the feature map to be detected will not change the size of the feature map, but can reduce the number of channels of the feature map to obtain a fused feature map with a single channel number.

[0057] The activation layer 122 uses a normalization function to normalize the values ​​in the fused feature map so that each value in the fused feature map falls within the interval [0, 1], thereby obtaining the background mask 210. For example, a Sigmoid activation function or a ReLU activation function can be used for normalization. The activation layer 122 does not change the size of the feature map, so the size of the background mask 210 input to the activation layer 122 is equal to the size of the fused feature map, which is equal to the size of the feature map to be detected.

[0058] Return to see Figure 2 and Figure 3 , execute step S130, use the background mask 210 to remove the background features in the feature map to be detected, and obtain a defect feature map.

[0059] In some embodiments, step S130 may be implemented in the following manner: performing a dot multiplication operation on the background mask 210 and the feature map to be detected to obtain a defect feature map.

[0060] As mentioned above, the background mask 210 and the feature map to be detected have the same size, that is, the number of rows and columns of the matrices they correspond to are equal. The dot product operation is to multiply the elements at the same position in two matrices with the same number of rows and columns. For example, the matrix and matrix The dot product of Some of the feature information in the feature image to be detected is used to determine the defect category, while some is background information that is irrelevant to determining the defect category. Based on this, the feature information used to determine the defect category is defined as defect features, also known as foreground features, while the background information that is irrelevant to determining the defect category is defined as background features.

[0061] As shown above, the larger the value of the pixel in the background mask 210, the greater the defect probability of the pixel, and the smaller the value, the smaller the defect probability of the pixel. Therefore, performing a dot multiplication operation on the background mask 210 and the feature map to be detected is equivalent to screening the features in the feature map to be detected. The background mask provides different coefficients for different values ​​in the feature map to be detected based on the degree of correlation with the determined defect category, and multiplies each value in the feature map to be detected by the corresponding coefficient, wherein the values ​​irrelevant to the determined defect category are reduced, and the values ​​strongly correlated with the determined defect category remain basically unchanged. In this way, the background features in the feature map to be detected can be removed, and the defect features related to the determined defect type are retained to obtain a defect feature map. In this step, the background mask 210 is used to remove background features irrelevant to the determined defect category to reduce interference with defect classification and improve classification accuracy.

[0062] Continue to see Figure 3 The defect detection model further includes a classification module 130, which receives a plurality of defect feature maps and determines a defect category of the PCB to be detected based on the plurality of defect feature maps.

[0063] For example, step S140, determining the defect category of the PCB to be inspected based on the defect feature map, may specifically include: obtaining a class activation map of each defect category in a plurality of defect categories and a class activation map of a background category based on the defect feature map;

[0064] Determine the probability that the PCB image to be detected corresponds to each category based on multiple class activation maps;

[0065] Based on the multiple probability values, the defect category of the PCB to be inspected is determined.

[0066] like Figure 3 As shown, classification module 130 includes a classification convolution block 131 and a labeling module 132. The defect feature map is input into classification convolution block 131 to generate C+1 class activation maps (CAMs), where C represents the number of defect categories, is a positive integer, and 1 represents the background category. These C+1 CAMs include CAMs corresponding to the C defect categories and one CAM corresponding to the background category. These C+1 CAMs are input into labeling module 132, which determines the probability value of each category based on the CAM. Based on these multiple probability values, the defect category of the PCB to be inspected is then determined.

[0067] The number of defect categories, C, can be determined based on the actual defect categories. In one embodiment, the number of defect categories, C, can be 6, and the defect categories may include: missing hole, rat bite, open circuit, short circuit, burr, and fake copper. Correspondingly, the number of class activation maps is 7.

[0068] In this embodiment, the classification convolution block 131 is a convolution block for outputting a class activation map. The present disclosure does not limit the specific network structure of the classification convolution block 131. In some embodiments, the classification convolution block 131 may include multiple convolution layers and a global average pooling layer at the last level. The convolution layer performs feature learning on the defect feature map to obtain richer features. The global average pooling layer uses the multiple feature maps output by the last convolution layer to obtain a class activation map. For example, the last convolution layer outputs K feature maps, and each category has weights corresponding to the K feature maps (W1, W2...W k ), using the weighted summation of these K feature maps, we can obtain the class activation map corresponding to the category. The weight corresponding to each category can be determined through model training. The larger the value of the pixel in the class activation map, the greater the contribution of that location to determining the defect category.

[0069] For example, the size of the class activation map is equal to the size of the defect feature map. The classification convolution block 131 may also include a deconvolution layer, which is located between the last convolution layer and the global average pooling layer. The deconvolution layer is used to upsample the feature map output by the last convolution layer to restore the feature map to the same size as the defect feature map. The global average pooling layer is used to process the feature map output by the deconvolution layer. For example, the classification convolution block 131 may use a UNet-like structure and retain low-level detail features through residual connections.

[0070] In some embodiments, the step of determining the probability of each class based on multiple class activation maps may specifically include: selecting multiple maximum values ​​on each class activation map and calculating the average of the multiple values; and using the average as the probability of the class corresponding to the class activation map.

[0071] Here, selecting the largest multiple values ​​means sorting all values ​​on the class activation map from largest to smallest and selecting the top few values. For example, the top K algorithm can be used to select the top K values ​​from the class activation map. For example, if the class activation map is 224×224, the top K values ​​are selected from the 224×224 values. The average of these K values ​​is then calculated and used as the probability of the class corresponding to the class activation map.

[0072] It should be noted that, in some other embodiments, other algorithms may be used to determine the probability of the defect category from the class activation map. For example, the average value of all values ​​on the class activation map may be calculated.

[0073] In some embodiments, the step of determining the defect category of the PCB to be inspected based on multiple probability values ​​may specifically include: setting a classification threshold, comparing the multiple probability values ​​with the classification threshold; and determining the defect category of the PCB to be inspected based on the comparison result.

[0074] For example, based on the comparison result, determining the defect category of the PCB image to be inspected includes: determining that the category corresponding to the probability value greater than or equal to the classification threshold is the defect category of the PCB to be inspected; if multiple probability values ​​are all less than the classification threshold, determining that the PCB image to be inspected has no defects.

[0075] This disclosure does not limit the specific value of the classification threshold, which can be set based on actual circumstances. In actual testing, there may be one or more probability values ​​that reach the classification threshold, thereby determining whether the PCB under inspection has one or more defects, and also determining the specific category of the defect.

[0076] In some embodiments, the classification module 130 can output the defect category of the PCB to be inspected in the form of a classification label. The defect detection model provided by the embodiments of the present disclosure uses image-level labels, and the final output of the classification module 130 is a label vector with a dimension of 1×1×(C+1). For example, if there are 6 defect categories, the dimension of the label vector is 1×1×7. For a PCB image to be inspected with only one defect, the value corresponding to the defect category in the output label vector is 1, and the other values ​​are 0. For example, for a PCB image to be inspected with only one "missing hole", the label vector can be [1, 0, 0, 0, 0, 0].

[0077] In some embodiments, the PCB defect detection method provided by the embodiments of the present disclosure further includes: locating a defective area of ​​the PCB to be detected based on the defect feature map.

[0078] For example, the step of locating the defective area of ​​the PCB to be inspected based on the defect feature map may specifically include:

[0079] A class activation map corresponding to the defect category of the PCB image to be inspected is selected; multiple candidate boxes are determined in the selected class activation map based on a set of positioning thresholds; and a bounding box of the defect area of ​​the PCB to be inspected is determined from the multiple candidate boxes to locate the defect area of ​​the PCB to be inspected.

[0080] In this embodiment, in order to locate the defect area from the selected class activation map, the class activation map can be segmented using a thresholding technique. For example, a set of positioning thresholds can be used to select connected areas to obtain multiple candidate boxes. For example, the positioning threshold can be set to [0.2, 0.6] of the maximum value in the class activation map, with a step size of 0.05. That is, the positioning threshold can be set to 20% to 60% of the maximum value in the class activation map, with a step size of 5%. Based on this, 9 candidate boxes can be obtained, and each positioning threshold corresponds to a candidate box. Taking the positioning threshold of 0.2 as an example, an area with a value greater than 20% of the maximum value in the class activation map is selected to obtain a candidate box. Based on this set of positioning thresholds, multiple candidate boxes can be obtained, and the area selected by each candidate box is a connected area.

[0081] After obtaining multiple candidate boxes, the non-maximum suppression algorithm (NMS) or the soft non-maximum suppression algorithm (softNMS) can be used to determine the bounding box of the defect area from the multiple candidate boxes. The NMS algorithm can post-process multiple candidate boxes, remove candidate boxes with high overlap, and retain only the candidate box with the highest prediction score as the detection output. The specific operation is as follows:

[0082] First, we need to obtain the location information and predicted probability of each candidate box, and then determine whether the overlapping area between the candidate box with the maximum probability and the candidate box it overlaps is greater than the set value. The overlapping area is determined by the following formula:

[0083]

[0084] When the overlapping area exceeds a set value, the candidate box with the highest probability is retained, and the others are deleted. The remaining candidate boxes are then further evaluated for overlapping area and deleted. This process is repeated until all predicted boxes have been found. The predicted probability of a candidate box is the probability that the candidate box belongs to the defect category determined above. Non-maximum suppression is used to find the candidate box with the highest probability and output it as the bounding box. The area within the bounding box is the defect area determined by the defect detection model.

[0085] The softNMS algorithm reduces the confidence of candidate boxes whose IOU is greater than a threshold, rather than retaining the candidate box with the highest probability as the bounding box and deleting the other candidate boxes. Compared with the NMS algorithm, the softNMS algorithm can reduce missed detections.

[0086] The disclosed embodiments provide a method for PCB defect detection. This method utilizes background modeling to generate a background mask. Based on this background mask, background features are removed from a feature map to be detected, ensuring that only the features relevant to determining the PCB defect category are retained. This reduces the interference of background features on the determination of PCB defect categories and improves the accuracy of PCB defect category detection and classification in complex image backgrounds. Furthermore, this defect detection model has a small number of parameters, and the defect feature map obtained after background feature removal contains less feature information than the feature map to be detected. This reduces the computational complexity of subsequent feature learning and classification of the defect feature map, resulting in fast inference speed and meeting the real-time requirements of industrial deployment.

[0087] The embodiments of the present disclosure also provide a defect detection model training method for training the above-mentioned defect detection model to determine the parameters in the model. Figure 6 A flow chart of a method for training a defect detection model provided by an embodiment of the present disclosure is shown as follows: Figure 6 As shown, the defect detection model training method includes:

[0088] S210: performing scale transformation processing on the original scale defect image to obtain a transformed scale defect image;

[0089] S220: Label the defect categories of the original scale defect image and the transformed scale image respectively to obtain an original sample image and a transformed sample image;

[0090] S230: Obtaining a first background mask based on the original sample image, and obtaining a second background mask based on the transformed sample image;

[0091] S240: Determine a loss function of the defect detection model based on the first background mask and the second background mask.

[0092] In the disclosed embodiments, the original scale defect image includes an original defect image, which is an original image captured by an image acquisition device. For example, the original defect image is an original PCB image captured by an industrial camera, which includes a CCD camera and / or a CMOS camera in an automatic optical inspection device.

[0093] The defect category can be determined according to production needs. For example, Figures 7a to 7f Multiple PCB defect images are provided, and the area within the bounding box in the image is the defect area. Figure 7a Shows the hole defect, Figure 7b A mouse-bite defect is shown, which is a loss of edge on the metal trace. Figure 7c shows an open circuit defect, Figure 7d shows a short circuit defect, Figure 7e shows a burr defect, Figure 7fA certain number of PCB defect images can be obtained for each defect category to establish a PCB defect dataset.

[0094] In some embodiments, the original scale defect image also includes a preprocessed defect image, and the method further includes: performing a preprocessing operation on the original defect image to obtain a preprocessed defect image; wherein the preprocessing operation includes at least one of brightness adjustment, contrast adjustment, rotation, flipping and block perturbation.

[0095] The purpose of preprocessing the original defect images is to expand the PCB defect dataset. The PCB defect dataset can be divided into a training set and a test set. The training set is used to train the defect detection model to determine its parameters, while the test set is used to test whether the defect detection model can accurately detect PCB defects.

[0096] During preprocessing, brightness adjustment includes randomly increasing or decreasing the brightness of the original defect image. Contrast adjustment can also be performed on the original defect image, such as randomly increasing or decreasing the contrast of the original defect image. Flipping the original defect image includes flipping it horizontally or vertically. Rotating the original defect image includes rotating it around an axis perpendicular to its plane at any angle, such as 90°, 100°, 130°, 180°, 230°, 270°, or 320°.

[0097] Performing block-based perturbation on the original defect image involves randomly segmenting the original defect image into multiple blocks and transforming the segmented blocks. For example, random transformations on the segmented blocks include randomly increasing or decreasing the distance between adjacent blocks, randomly adjusting the position of blocks, randomly flipping or rotating blocks, etc. Block-based perturbation is targeted for PCB defect detection. This is because different regions of a PCB are less correlated, unlike other image tasks that have rich high-level semantic information. Training on block-based perturbation input images can improve the accuracy of the model's PCB defect detection.

[0098] The following combination Figures 8a to 8e Explain the preprocessing of the original defect image. Figure 8a shows a schematic diagram of an original defect image, Figure 8b The preprocessed defect image obtained by random brightness enhancement of the original defect image is shown. Figure 8c The pre-processed defect image obtained by horizontally flipping the original defect image is shown. Figure 8d The preprocessed defect image obtained by randomly rotating the original defect image is shown. Figure 8e and Figure 8fThe different pre-processed defect images obtained after different block perturbations are performed on the same original defect image are shown. Figure 8e and Figure 8f In this paper, the original defect image is segmented into 16 blocks, and these blocks are transformed by position shifting, distance adjustment, flipping, and rotation. Because the sizes of the different blocks vary after segmentation, the transformations to the blocks are different. Therefore, a large number of preprocessed defect images can be obtained by randomly perturbing the same original defect image.

[0099] See also Figure 6 , executing step S210: performing a resizing process on the original scale defect image to obtain a resizing image. Here, resizing includes randomly increasing or decreasing the size of the original scale defect image. For example, the width and height of the original scale defect image may be reduced to half, or the width and height of the original scale defect image may be doubled.

[0100] In some embodiments, bilinear interpolation may be used to perform random scale transformation. For example, a transformation threshold may be pre-set, such as [0.5, 1.5].

[0101] Continue to see Figure 6 , execute step S220, respectively label the defect categories of the original scale defect image and the transformed scale image to obtain the original sample image corresponding to the original scale defect image and the transformed sample image corresponding to the transformed scale defect image.

[0102] For example, the annotation of the defect image includes image-level category annotation of the defect image. Image-level category annotation refers to only annotating the category to which the relevant defects in the image belong, without selecting the defect area. In other embodiments, the defect image can also be annotated with an object frame, such as Figures 7a to 7f As shown, mark the rectangular box where the defect is located and the defect category.

[0103] Next, step S230 is executed to obtain a first background mask based on the original sample image and a second background mask based on the transformed sample image.

[0104] For example, step S230 may be implemented as follows:

[0105] Performing feature extraction processing on the original sample image to obtain an original sample feature map corresponding to the original sample image; performing background modeling processing based on the original sample feature map to obtain a first background mask corresponding to the original sample feature map;

[0106] Perform feature extraction processing on the transformed sample image to obtain a transformed sample feature map corresponding to the transformed sample image; perform background modeling processing based on the transformed sample feature map to obtain a second background mask corresponding to the transformed sample feature map.

[0107] For example, the module that performs feature extraction on the original sample image and the module that performs feature extraction on the transformed sample image share parameters, and the module that performs background modeling processing on the original sample feature map and the module that performs background modeling processing on the transformed sample feature map share parameters.

[0108] Figure 9 Schematic diagram of another defect detection model provided by the embodiment of the present disclosure. Figure 9 As shown, the defect detection model can include an original scale branch and a transformed scale branch. The original scale branch's input is the original sample image, while the transformed scale branch's input is the transformed sample image. Each scale branch further includes a base branch and a background filtering branch. The background filtering branch in the original scale branch includes a first feature extraction module 310, a first filtering module 320, and a first classification module 330. The base branch in the original scale branch includes the first feature extraction module 310, which is shared with the background filtering branch, and a second classification module 340.

[0109] During the defect detection model training phase, in the original scale branch, the first feature extraction module 310 is used to perform feature learning on the original sample image to obtain an original sample feature map. For example, the original sample feature map and the original sample image have the same size, and there can be multiple original sample feature maps.

[0110] For example, the first feature extraction module 310 includes a convolution block using a fully convolutional network. The fully convolutional network may include multiple convolutional layers and deconvolutional layers, but no fully connected layers. The first feature extraction module 310 obtains an original sample image and performs a convolution operation on the original sample image to obtain an original sample feature map. For example, the convolution block using the fully convolutional network may use a UNet structure and may also retain low-level detail features through residual connections.

[0111] The first filtering module 320 is used to perform background modeling processing based on the original sample feature map to obtain a first background mask 211. The size of the first background mask 211 is equal to the size of the original sample feature map. For example, the first filtering module 320 includes a first feature fusion layer and a first activation layer. The original sample feature map is input into the first feature fusion layer for feature fusion processing to obtain an original fused feature map. The original fused feature map and the original sample feature map have the same size. Next, the original fused feature map is input into the first activation layer for normalization processing to obtain the first background mask 211. For example, the first feature fusion layer includes a 1×1 convolution layer.

[0112] Continue to see Figure 9 After obtaining the first background mask 211, the method further includes:

[0113] removing background features in the original sample feature map based on the first background mask 211 to obtain an original sample defect feature map; determining a first set of probability values ​​corresponding to multiple categories of the original sample image based on the original sample defect feature map; and calculating a first cross entropy loss function based on the first set of probability values;

[0114] Based on the original sample feature map, a second set of probability values ​​corresponding to multiple categories of the original sample image is determined, and a second cross entropy loss function is calculated based on the second set of probability values; the loss function includes a first cross entropy loss function and a second cross entropy loss function.

[0115] Similar to the above-mentioned step of using the background mask 210 to remove the background features in the feature map to be detected to obtain the defect feature map, the first background mask 211 and the original sample feature map can be multiplied by a dot product operation to obtain the original sample defect feature map.

[0116] The first classification module 330 includes a first classification convolution block 331 and a first labeling module 332. The first classification convolution block 331 is used to obtain C+1 class activation maps based on the original sample defect feature map, where C represents the number of defect categories and 1 represents the background category. The C+1 class activation maps include C class activation maps corresponding to the C defect categories and one class activation map corresponding to the background category. The first labeling module 332 can also obtain a first set of probability values ​​for multiple categories based on the C+1 class activation maps output by the first classification convolution block 331. For example, a Top K algorithm can be used to select the largest K values ​​in each class activation map and average the K values ​​to obtain the probability of the defect category corresponding to the class activation map.

[0117] In addition, the first label module 332 can generate a first classification label based on the first set of probability values. The defect detection model provided by the embodiment of the present disclosure uses image-level labels, and the final output is a label vector with a dimension of 1×1×(C+1). For example, if there are 6 defect categories, the dimension of the label vector is 1×1×7. For example, for an original sample image with only "missing hole", the first classification label can be [1, 0, 0, 0, 0, 0]. For an original sample image including "missing hole" and "rat bite", the first classification label can be [1, 0, 1, 0, 0, 0, 0].

[0118] Continue to see Figure 9 The first classification module 330 may further include a first function module 333, the first function module 333 is used to calculate a first cross entropy loss function according to a first set of probability values ​​corresponding to multiple categories The loss function is used to measure the degree of deviation between the prediction results made by the defect detection model and the actual results. A smaller loss function value indicates that the model's prediction results are closer to the actual results, and vice versa, it indicates a larger error. During the defect detection model training phase, the parameters in the model can be adjusted based on the value of the loss function to improve the accuracy of the model's prediction results. In the embodiment of the present disclosure, the loss function used to measure the defect detection model of the present disclosure includes a first cross entropy loss function

[0119] In some embodiments, as Figure 9 As shown, the original scale branch also includes a basic branch, which shares the first feature extraction module 310 with the background filtering branch. The basic branch also includes a second classification module 340, which includes a second classification convolution block 341, a second label module 342, and a second function module 343. The second classification convolution block 341 is connected to the first feature extraction module 310, and is used to receive the original sample feature map and obtain C+1 class activation maps based on the original sample feature map. The number of class activation maps output by the second classification convolution block 341 and the first classification convolution block 331 is the same, but the difference is that the class activation map output by the second classification convolution block 341 includes background feature information. The second classification convolution block 341 and the first classification convolution block 331 share parameters, that is, the convolution kernels of the two convolution blocks are exactly the same and the weights when obtaining the class activation map are shared.

[0120] The second label module 342 is used to obtain a second set of probability values ​​corresponding to multiple categories of the original sample image based on the C+1 class activation maps output by the second classification convolution block 341; and to generate a second classification label based on the second set of probability values. For example, for the original sample image with only "holes", the second classification label is [1, 0, 0, 0, 0, 0, 1], where the last value of the label represents the background category. Since the input of the second classification convolution block 341 is the original sample feature map and the background features are not filtered, the value corresponding to the background category in the second classification label is 1. The second label module 342 and the first label module 332 use the same algorithm to process the input class activation map to output the classification label.

[0121] The second function module 343 is used to calculate the second cross entropy loss function according to the second set of probability values ​​corresponding to multiple categories The loss function used to measure the defect detection model of the present disclosure includes the second cross entropy loss function

[0122] In this embodiment, by designing a basic branch and a background filtering branch, and obtaining class activation maps for both branches to calculate the loss function, the purpose is to utilize the twin network structure to construct contrastive learning to improve the scale robustness of the defect detection model.

[0123] In some embodiments, the cross entropy loss function of the base branch in the different scale branches is The calculation formula is:

[0124]

[0125] Cross entropy loss function of background filtering branches in different scale branches The calculation formula is:

[0126]

[0127] In the above cross-entropy loss function, C represents the number of defect categories, and c is a positive integer between 1 and C+1. N represents the number of original sample images, and n is a positive integer between 1 and N. i represents the scale branch, where i is 1 for the original scale branch.

[0128] Represents the true probability value of the cth category among multiple categories corresponding to the nth original sample image in the basic branch, Represents the predicted probability value of the cth category among multiple categories corresponding to the nth original sample image in the basic branch.

[0129] Represents the true probability value of the cth category among multiple categories corresponding to the nth transformed sample image in the background filtering branch, Represents the predicted probability value of the cth category among multiple categories corresponding to the nth transformed sample image in the background filtering branch. The first cross entropy loss function The second cross entropy loss function is calculated using formula (2) and formula (3).

[0130] Continue to see Figure 9 The scale-transformation branch and the original scale-transformation branch have the same network structure. The scale-transformation branch also includes a base branch and a background filtering branch. The background filtering branch includes a second feature extraction module 410, a second filtering module 420, and a third classification module 430. The base branch includes the second feature extraction module 410 shared with the background filtering branch, and a fourth classification module 440.

[0131] The second feature extraction module 410 is used to perform feature learning on the transformed sample image to obtain a transformed sample feature map, the size of which is equal to the size of the transformed sample image. The second filtering module 420 performs background modeling processing based on the transformed sample feature map to obtain a second background mask 212, the size of which is equal to the size of the transformed sample feature map.

[0132] For example, the second feature extraction module 410 includes a convolution block using a fully convolutional network. The second feature extraction module 410 and the first feature extraction module 310 share parameters. Here, the shared parameters include: the convolution kernel in the shared convolution block, that is, the convolution kernels of the convolution blocks in the two scale branches are exactly the same. The second filtering module 420 includes a second feature fusion layer (such as a 1×1 convolution layer) and a second activation layer. The second feature fusion layer and the first feature fusion layer share parameters, that is, the convolution kernel of the shared convolution layer. The second activation layer uses the same activation function as the first activation layer. For example, both activation layers use the Sigmoid activation function.

[0133] In the scale transformation branch, after obtaining the second background mask 212, the method further includes:

[0134] removing background features in the transformed sample feature map based on the second background mask 212 to obtain a transformed sample defect feature map; determining a third set of probability values ​​that the transformed sample image corresponds to multiple categories based on the transformed sample defect feature map; and calculating a third cross entropy loss function based on the third set of probability values;

[0135] Based on the transformed sample feature map, a fourth set of probability values ​​corresponding to multiple categories of the transformed sample image is determined, and a fourth cross entropy loss function is calculated based on the fourth set of probability values; the loss function also includes a third cross entropy loss function and a fourth cross entropy loss function.

[0136] For example, removing background features in the transformed sample feature map based on the second background mask 212 to obtain the transformed sample defect feature map includes: performing a dot product operation on the second background mask 212 and the transformed sample feature map to obtain the transformed sample defect feature map.

[0137] like Figure 9 As shown, the third classification module 430 includes a third classification convolution block 431, a third labeling module 432, and a third function module 433, which are sequentially connected. The second transformation branch module is used to obtain C+1 class activation maps based on the transformed sample defect feature map. The third labeling module 432 obtains a third set of probability values ​​corresponding to multiple classes of the transformed sample image based on the C+1 class activation maps output by the third classification convolution block 431. The third function module 433 calculates a third cross-entropy loss function based on the third set of probability values. Furthermore, the third labeling module 432 can generate a third classification label based on the third set of probability values.

[0138] The fourth classification module 440 includes a fourth classification convolution block 441, a fourth labeling module 442, and a fourth function module 443, which are sequentially connected. The fourth classification convolution block 441 is configured to obtain C+1 class activation maps based on the transformed sample feature map. The fourth labeling module 442 obtains a fourth set of probability values ​​corresponding to multiple classes of the transformed sample image based on the C+1 class activation maps output by the fourth classification convolution block 441. The fourth function module 443 calculates a fourth cross-entropy loss function based on the fourth set of probability values. Furthermore, the fourth labeling module 442 can generate a fourth classification label based on the fourth set of probability values.

[0139] For example, Figure 9 , the first classification convolution block 331 , the second classification convolution block 341 , the third classification convolution block 431 and the fourth classification convolution block 441 share parameters, that is, they share convolution kernels and weights.

[0140] For example, the first labeling module 332 , the second labeling module 342 , the third labeling module 432 and the fourth labeling module 442 use the same algorithm, for example, they all use the Top K algorithm first and then calculate the average value to obtain the probability value corresponding to each category.

[0141] For example, the third cross entropy loss function can be calculated using the above formula (2), and the fourth cross entropy loss function can be calculated using the above formula (3).

[0142] In some embodiments, the number of transform scaling branches may be 1, such as Figure 9 As shown. In other embodiments, the number of transform scale branches can be multiple, and the inputs of different transform scale branches are transformed scale defect images of different sizes obtained by transforming the same original scale defect image at different scales. Among them, the cross entropy loss function of the base branch in the different scale branches is calculated using the above formula (2), and the cross entropy loss function of the background filter branch in the different scale branches is calculated using the above formula (3). It should be understood that for the same original scale defect image, the base branches in the different scale branches can share a set of true probability values, and the background filter branches in the different scale branches can share a set of true probability values.

[0143] In the disclosed embodiment, the parameters and weights of the scale transformation branch and the original scale branch are shared. In essence or in terms of code implementation, the scale transformation branch inputs the transformed sample image into the network of the original scale branch. However, the image scale is different, and the size of the feature map will be different during the process.

[0144] The following combination Figure 9A specific embodiment is used for illustration. For example, the dimensions of the original sample image are 224×224×3, and the dimensions of the transformed sample image obtained by scaling the original sample image are 112×112×3, that is, the size of the original sample image is halved.

[0145] In the original scale branch, the original sample feature map obtained after the original sample image passes through the first feature extraction module 310 is 224×224×256, the dimension of the first background mask 211 is 224×224×1, the dimension of the class activation map after passing through the first classification convolution block 331 or the second classification convolution block 341 is 224×224×7, the first label module 332 or the second label module 342 obtains 7 probability values, and the dimension of the output classification label is 1×1×7.

[0146] In the transformation scale branch, the transformed sample feature map of the transformed sample image after passing through the second feature extraction module 410 is 112×112×256, the dimension of the second background mask 212 is 112×112×1, and the dimension of the class activation map after passing through the third classification convolution block 431 or the fourth classification convolution block 441 is 112×112×7. The third label module 432 or the fourth label module 442 obtains 7 probability values, and the dimension of the output classification label is 1×1×7.

[0147] In some embodiments, the loss function further includes a scale consistency loss function and a normalization loss function. In step S240, the loss function of the defect detection model is calculated based on the first background mask and the second background mask, including:

[0148] Calculating a scale consistency loss function based on the first background mask and the second background mask;

[0149] A normalized loss function is calculated based on a plurality of first background masks corresponding to a plurality of original sample images and a plurality of second background masks corresponding to a plurality of transformed sample images.

[0150] For example, the calculation formula of the scale consistency loss function is:

[0151] L sim =||θ(M1)-M2||1

[0152] Here, θ(M1) and M2 are both matrices. θ(M1) is obtained by performing the same scale transformation on the matrix M1 corresponding to the first background mask as that used to transform the original sample image to the transformed sample image, so that θ(M1) and the matrix M2 corresponding to the second background mask are equal in size. The scale consistency loss function is the L1 norm of the new matrix obtained by subtracting θ(M1) and M2, which is used to measure the robustness of the defect detection module.

[0153] For example, the calculation formula of the normalized loss function is:

[0154]

[0155] Among them, when using the original scale branch to calculate the normalized loss function, N represents the number of original sample images, |W n | represents the value of the first background mask 211, which is a matrix, |W n | is the value of the matrix. When using the transformation scale branch to calculate the normalized loss function, N represents the number of transformed sample images, which is equal to the number of original sample images; |W n | represents the value of the second background mask 212. The normalized loss function, also known as the sparsity loss, measures the sparsity of the model. Specifically, it aims to have more zero values ​​in the background mask to ensure that the background mask captures only defect features and not background features, thus preventing non-defect features from being identified as defect features. The sparse loss function helps reduce the computational complexity of the subsequent classification module and improves the efficiency of model training and prediction.

[0156] In this disclosure, the loss function used to measure the defect detection model includes the cross entropy loss function of the base branch of each scale branch. And the cross entropy loss function of the background filtering branch Scale consistency loss function and normalization loss function. For example, the loss function is obtained by performing a weighted sum operation on the aforementioned multiple types of loss functions, that is, the calculation formula of the loss function is:

[0157] L total =αL base +βL bg +γL sim +δL norm

[0158] Among them, the cross entropy loss function L of the base branch base is the cross entropy loss function of the base branch of each scale branch The sum of Cross entropy loss function L of the background filtering branch bg is the cross entropy loss function of the background filtering branch of each scale branch The sum of Normalized loss function L norm is the sum of the normalized loss functions of the original scale branch and the transformed scale branch, that is,

[0159] In this disclosed embodiment, the original scale branch and the background scale branch form a twin network. Through parameter and weight sharing, similarity comparison learning of background masks (i.e., scale consistency loss), cross-entropy loss functions for different scales and branches, and a sparsity loss function, the defect detection model achieves improved scale robustness. Furthermore, this model has a small number of parameters and computational complexity, and boasts fast inference speed, meeting the real-time requirements of industrial deployment.

[0160] In some embodiments, the training method further includes inputting test sample images from a test set into a background filtering branch within an original-scale branch of a defect detection model for testing to verify the accuracy of the defect detection model. The test set may include a portion of original-scale defect images or transformed-scale defect images partitioned from a PCB defect dataset.

[0161] In this disclosure, the use Figure 9 The background filter branch in the original scale branch of the defect detection model shown in the figure is used for inference, that is, the image of the PCB to be detected is input into the background filter branch in the original scale branch for prediction to determine the defect category of the PCB to be detected. Figure 2 The feature extraction module 110, the filtering module 120 and the classification module 130 correspond to Figure 9 The defect detection model shown includes the first feature extraction module 310, the first filtering module 320, and the first classification module 330. Essentially, the rescaling branch directly inputs the rescaled sample image into the original rescaling branch's network. However, due to the varying sizes of the input images, the resulting feature maps vary in size. The defect detection model can be considered to consist solely of the first feature extraction module 310, the first filtering module 320, the first classification module 330, and the second classification module 340. The rescaling of the original sample image is primarily to expand the sample dataset and to introduce a multi-scale contrastive learning mechanism to enhance the model's scale robustness and further improve its detection accuracy.

[0162] An embodiment of the present application provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the PCB defect detection method or defect detection model training method provided in this application.

[0163] An embodiment of the present application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the PCB defect detection method provided by the embodiment of the present disclosure, or execute the defect detection model training method provided by the embodiment of the present disclosure.

[0164] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0165] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0166] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0167] The present disclosure also provides an electronic device, Figure 10 A schematic diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 10 As shown, the electronic device 500 includes a memory 510, a processor 520, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the PCB defect detection method described above, or to implement the defect detection model training method described above.

[0168] For example, the electronic device 500 may be Figure 1 The computer in the automatic optical inspection device can be used to run a computer program for implementing the PCB defect detection method or defect detection model training method provided by the present disclosure in order to detect the defect type of the PCB online and in real time.

[0169] In summary, the disclosed embodiments utilize weakly supervised learning and data augmentation methods to reduce the data requirements of visual inspection systems, demonstrating good generalization. This detection method, based on a background modeling network, improves classification and detection accuracy in complex image backgrounds and incorporates a multi-scale contrastive learning mechanism to enhance the model's scale robustness. Furthermore, the detection model features a small number of parameters and computational complexity, resulting in fast inference speed, meeting the real-time requirements of industrial deployment.

[0170] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed herein shall be covered by the claims of this disclosure.

Claims

1. A defect detection model training method, characterized in that: include: Performing scale transformation processing on the original scale defect image to obtain a transformed scale defect image; The defect categories of the original scale defect image and the transformed scale image are respectively labeled to obtain the original sample image and the transformed sample image; Obtaining an original sample feature map and a first background mask based on the original sample image, and obtaining a second background mask based on the transformed sample image; Determining a scale consistency loss function and a normalization loss function of the defect detection model based on the first background mask and the second background mask; removing background features in the original sample feature map based on the first background mask to obtain an original sample defect feature map; determining a first set of probability values ​​corresponding to multiple categories of the original sample image based on the original sample defect feature map; Calculating a first cross entropy loss function based on the first set of probability values; wherein the multiple categories include multiple defect categories and a background category; Based on the original sample feature map, determine a second set of probability values ​​that the original sample image corresponds to the multiple categories, and calculate a second cross entropy loss function based on the second set of probability values; the loss function includes the scale consistency loss function, the normalization loss function, the first cross entropy loss function and the second cross entropy loss function.

2. The defect detection model training method according to claim 1, characterized in that: The obtaining of an original sample feature map and a first background mask based on the original sample image, and obtaining a second background mask based on the transformed sample image, comprises: Performing feature extraction processing on the original sample image to obtain an original sample feature map corresponding to the original sample image; performing background modeling processing based on the original sample feature map to obtain a first background mask corresponding to the original sample feature map; Performing feature extraction processing on the transformed sample image to obtain a transformed sample feature map corresponding to the transformed sample image; and performing background modeling processing based on the transformed sample feature map to obtain a second background mask corresponding to the transformed sample feature map.

3. The defect detection model training method according to claim 2, characterized in that: The feature extraction process includes: The feature extraction process is performed using a fully convolutional network.

4. The defect detection model training method according to claim 2, characterized in that: The performing background modeling processing based on the original sample feature map to obtain a first background mask corresponding to the original sample feature map includes: performing feature fusion processing on the original sample feature map using a feature fusion layer to obtain a first fused feature map, and performing normalization processing on the first fused feature map using an activation layer to obtain the first background mask; The background modeling processing is performed based on the transformed sample feature map to obtain a second background mask corresponding to the transformed sample feature map, including: using the feature fusion layer to perform feature fusion processing on the transformed sample feature map to obtain another fused feature map, and using the activation layer to normalize the other fused feature map to obtain the second background mask.

5. The defect detection model training method according to claim 2, characterized in that: The method further comprises: removing background features in the transformed sample feature map based on the second background mask to obtain a transformed sample defect feature map; determining a third set of probability values ​​for the transformed sample image corresponding to the multiple categories based on the transformed sample defect feature map; and calculating a third cross entropy loss function based on the third set of probability values; Based on the transformed sample feature map, determine a fourth set of probability values ​​that the transformed sample image corresponds to the multiple categories, and calculate a fourth cross entropy loss function based on the fourth set of probability values; the loss function also includes the third cross entropy loss function and the fourth cross entropy loss function.

6. The defect detection model training method according to claim 5, characterized in that: The removing background features in the original sample feature map based on the first background mask to obtain the original sample defect feature map includes: performing a dot multiplication operation on the first background mask and the original sample feature map to obtain the original sample defect feature map; The removing background features in the transformed sample feature map based on the second background mask to obtain the transformed sample defect feature map includes: performing a dot multiplication operation on the second background mask and the transformed sample feature map to obtain the transformed sample defect feature map.

7. The defect detection model training method according to claim 5, characterized in that: Determining, based on the original sample defect feature map, a first set of probability values ​​corresponding to a plurality of categories of the original sample image includes: obtaining a first set of class activation maps of a plurality of categories based on the original sample defect feature map; and determining the first set of probability values ​​based on the first set of class activation maps; Determining, based on the original sample feature map, a second set of probability values ​​for the original sample image corresponding to the multiple categories includes: obtaining a second set of class activation maps for the multiple categories based on the original sample feature map; and determining the second set of probability values ​​based on the second set of class activation maps.

8. The defect detection model training method according to claim 7, characterized in that: Determining, based on the transformed sample defect feature map, a third set of probability values ​​that the transformed sample image corresponds to the multiple categories includes: obtaining a third set of class activation maps for the multiple categories based on the transformed sample defect feature map; and determining the third set of probability values ​​based on the third set of class activation maps; Determining, based on the transformed sample feature map, a fourth group of probability values ​​that the transformed sample image corresponds to the multiple categories includes: obtaining a fourth group of class activation maps for the multiple categories based on the transformed sample feature map; and determining the fourth group of probability values ​​based on the fourth group of class activation maps.

9. The defect detection model training method according to claim 8, characterized in that: Determining a probability value based on any one of the first group of class activation maps, the second group of activation maps, the third group of activation maps, and the fourth group of class activation maps includes: A plurality of maximum values ​​are selected on the class activation map, and an average value of the plurality of values ​​is determined as a probability value of a class corresponding to the class activation map.

10. The defect detection model training method according to claim 9, characterized in that: The determining of a scale consistency loss function and a normalization loss function of the defect detection model based on the first background mask and the second background mask includes: Calculating the scale consistency loss function based on the first background mask and the second background mask; The normalized loss function is calculated based on a plurality of the first background masks corresponding to a plurality of the original sample images and a plurality of the second background masks corresponding to a plurality of transformed sample images.

11. The defect detection model training method according to claim 5, characterized in that: The method further comprises: generating a first classification label based on the first set of probability values; generating a second classification label based on the second set of probability values; generating a third classification label based on the third set of probability values; A fourth classification label is generated based on the fourth set of probability values.

12. The defect detection model training method according to claim 1, characterized in that: The original scale defect image includes an original defect image and a preprocessed defect image; the method further includes: A preprocessing operation is performed on the original defect image to obtain the preprocessed defect image; wherein the preprocessing operation includes at least one of brightness adjustment, contrast adjustment, rotation, flipping and block disturbance.

13. A defect detection method, characterized in that: The method comprises: The image of the PCB to be inspected is input into a defect detection model trained by the defect detection model training method according to any one of claims 1 to 12 to determine the defect category of the PCB to be inspected.

14. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to: implement the defect detection model training method according to any one of claims 1 to 12, or implement the defect detection method according to claim 13.

15. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the defect detection model training method according to any one of claims 1 to 12 is implemented, or the defect detection method according to claim 13 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the defect detection model training method according to any one of claims 1 to 12, or implements the defect detection method according to claim 13.

Images