PCB surface defect detection method and system

Abstract

The application discloses a PCB surface defect detection method and system. The method comprises the following steps: constructing a training set, a verification set and a test set based on collected PCB defect image information; preprocessing the training set to obtain a multi-defect synthetic image; constructing a target detection neural network; training the target detection neural network by using the multi-defect synthetic image to obtain a defect detection model; verifying the defect detection model on the verification set to obtain a comprehensive detection model; and testing the comprehensive detection model on the test set to obtain a PCB surface defect detection result image. The method can effectively increase sample diversity. The constructed target detection neural network can sufficiently focus on the connection between defects and simultaneously give feedback to an image generation network, thereby synthesizing more samples that are beneficial to improving detection precision, increasing defect detection precision, ensuring the authenticity of generated images, improving the generalization ability of the defect detection model and effectively alleviating the foreground-foreground imbalance problem.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of surface defect detection, and particularly relates to a PCB surface defect detection method and system. BACKGROUND

[0002] In the development of modern electronic information industry, a printed circuit board (PCB) has become an irreplaceable basic component, which has the important functions of connecting, integrating and controlling various electronic components, and has a lasting influence on the production and use of various electronic information products. For example, smart phones, portable tablet computers and the like used in our daily life cannot be separated from the use of PCBs. Due to the extremely high integration of PCBs, any problem in PCBs will affect the use of PCBs, and therefore, defect detection of PCBs is particularly important.

[0003] In recent years, the development of neural networks has been extremely rapid, and neural networks can efficiently perform image classification, target detection, semantic segmentation and the like. In the aspect of defect detection, neural networks are also widely used, such as steel defect detection, PCB defect detection and the like.

[0004] The existing technology only focuses on identifying the target itself, and seriously ignores the relationship between targets. When objects are located at different image positions, different scales or different categories, their quantities in different images are basically not the same, resulting in foreground-foreground imbalance of the detected results, and affecting the detection accuracy. SUMMARY

[0005] Therefore, the embodiments of the present disclosure provide a PCB surface defect detection method and system, which can improve the defect detection accuracy of PCBs.

[0006] In a first aspect, the embodiments of the present disclosure provide a PCB surface defect detection method, comprising the following steps:

[0007] A training set, a verification set and a test set are constructed according to a preset proportion based on collected PCB defect image information;

[0008] The training set is preprocessed to obtain a multi-defect synthetic image;

[0009] A target detection neural network is constructed;

[0010] The multi-defect synthetic image is used to train the target detection neural network to obtain a defect detection model;

[0011] The defect detection model is verified on the verification set;

[0012] Test the defect detection model on the test set to obtain a PCB surface defect detection result image.

[0013] Optionally, the training set, the verification set and the test set are constructed according to a preset proportion based on the collected PCB defect image information, and the method comprises the following steps of:

[0014] Collecting a PCB finished product image;

[0015] Classifying the PCB finished product image based on a preset defect type and performing label processing to obtain a PCB defect image information dataset;

[0016] Dividing the PCB defect image information dataset into the training set, the verification set and the test set according to a proportion of 8:1:1.

[0017] Optionally, the training set is preprocessed to obtain a multi-defect synthetic image, and the method comprises the following steps of:

[0018] Based on the training set, a feature extraction network is used to perform feature extraction on a first foreground mask and a first background image to obtain a first foreground mask feature image and a first background feature image.

[0019] A synthetic network is used to process the first foreground mask feature image and the first background feature image to obtain the multi-defect synthetic image.

[0020] Optionally, the synthetic network is used to process the first foreground mask feature image and the first background feature image to obtain the multi-defect synthetic image, and the method comprises the following steps of:

[0021] The first foreground mask is processed in the same dimension to obtain a second foreground mask with the same dimension as the first background image.

[0022] An image processing method is used to fuse the second foreground mask and the first background image to obtain a first synthetic image.

[0023] A spatial transformation network is used to process the first background image, the second foreground mask and the first synthetic image to obtain a second synthetic image.

[0024] A first defect image with the same shape as the first foreground mask is extracted from a second background image.

[0025] The first defect image and the second synthetic image are fused to obtain a third synthetic image.

[0026] A first background mask with the same shape as the first defect image is extracted from the second background image.

[0027] fusing the first background mask and the third synthetic image to obtain a fourth synthetic image;

[0028] determining whether the fourth synthetic image contains at least three defects, and if yes, the fourth synthetic image is the multi-defect synthetic image;

[0029] if no, performing a loop step.

[0030] Optionally, the loop step comprises:

[0031] extracting an nth defect image in the mth background image, the nth defect image having the same shape as the nth foreground mask; wherein m is greater than or equal to 3, and n is equal to m-1;

[0032] fusing the nth defect image and the st synthetic image to obtain an st+1 synthetic image; wherein s is equal to 2m-2;

[0033] extracting an nth background mask in the mth background image, the nth background mask having the same shape as the nth defect image;

[0034] fusing the nth background mask and the st+1 synthetic image to obtain an st+2 synthetic image;

[0035] determining whether the st+2 synthetic image contains at least three defects, and if yes, the st+2 synthetic image is the multi-defect synthetic image,

[0036] if no, looping the above steps until the st+2 synthetic image contains at least three defects.

[0037] Optionally, the method further comprises the following steps:

[0038] performing secondary discrimination on the multi-defect synthetic image by using a discriminator;

[0039] determining whether the output true probability value is greater than a preset value, and if yes, determining that the multi-defect synthetic image is a real synthetic image;

[0040] if no, determining that the multi-defect synthetic image is a false synthetic image, triggering the synthetic network to change a training parameter, and regenerating a multi-defect synthetic image;

[0041] the training parameter comprises a learning rate and a loss function.

[0042] Optionally, the target detection neural network comprises convolution layers, pooling layers and fully connected layers connected in sequence;

[0043] the fully connected layers comprise a correlation module and a deletion network module, and the correlation module is used for performing correlation processing between different defects of the multi-defect synthetic image;

[0044] The deleting network module is configured to delete unsuitable regions of interest to obtain a preset set of regions of interest.

[0045] In a second aspect, the embodiments of the present disclosure provide a PCB surface defect detection system, comprising:

[0046] The dividing module is configured to construct a training set, a verification set and a test set according to a preset proportion based on the collected PCB defect image information.

[0047] The preprocessing module is configured to preprocess the training set to obtain a multi-defect synthetic image.

[0048] The constructing module is configured to construct a target detection neural network.

[0049] The training module is configured to train the target detection neural network using the multi-defect synthetic image to obtain a defect detection model.

[0050] The verification module is configured to verify the defect detection model on the verification set.

[0051] The test module is configured to test the defect detection model on the test set to obtain a PCB surface defect detection result image.

[0052] In a third aspect, the embodiments of the present disclosure further provide an electronic device, which adopts the following technical scheme:

[0053] The electronic device comprises:

[0054] at least one processor; and

[0055] a memory in communication with the at least one processor; wherein

[0056] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the PCB surface defect detection method described above.

[0057] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, which stores computer instructions for causing a computer to execute the PCB surface defect detection method described above.

[0058] The PCB surface defect detection method disclosed in the application, the obtained multi-defect synthetic image contains multiple types of defects, effectively increasing sample diversity; through the constructed target detection neural network, the connection between defects can be fully focused on, and feedback is given to the image generation network, more samples that are beneficial to improve detection accuracy are synthesized, the accuracy of defect detection is increased, the authenticity of the generated image is ensured, the generalization ability of the defect detection model is improved, and the foreground-foreground imbalance problem is effectively alleviated.

[0059] The application effectively solves the single defect problem of the image set and the foreground-foreground imbalance problem caused by the inability to use the connection between defects by adopting a synthetic network, effectively increases sample diversity, makes the image contain multiple defect features, and facilitates the use of information between defects.

[0060] The above description is only a summary of the technical solutions of the disclosure, in order to more clearly understand the technical means of the disclosure, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0062] Figure 1 The flowchart of the PCB surface defect detection method provided by the embodiments of the disclosure is shown.

[0063] Figure 2 The training set construction flowchart of the PCB surface defect detection method provided by the application is shown.

[0064] Figure 3 The flowchart of the multi-defect synthetic image acquisition method in the embodiments of the disclosure is shown.

[0065] Figure 4 The flowchart of the feature extraction synthesis in the embodiments of the disclosure is shown.

[0066] Figure 5 The flowchart of the cycle step in the embodiments of the disclosure is shown.

[0067] Figure 6 The schematic diagram of the target detection neural network in the embodiments of the disclosure is shown.

[0068] Figure 7Flowchart of another embodiment of the PCB surface defect detection method disclosed in the present application.

[0069] Figure 8 Flowchart of the acquisition process of the real synthetic image in the present application.

[0070] Figure 9 Principle block diagram of the PCB surface defect detection system provided by the embodiment of the present disclosure;

[0071] Figure 10 Principle block diagram of an electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0073] It should be apparent that the following describes the embodiments of the present disclosure through specific, concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all. The present disclosure can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0074] It should be noted that the various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, the devices and / or methods can be implemented using any number of the aspects set forth herein. In addition, this device and / or method can be implemented using other structures and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0075] It should also be noted that the drawings provided in the following embodiments only schematically illustrate the basic concept of the present disclosure, and only the components related to the present disclosure are shown in the drawings, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in shape, number and proportion, and the layout pattern of the components may also be more complex.

[0076] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the relevant art will appreciate that the described aspects can be practiced without these specific details.

[0077] Referring to Figure 1 A first aspect of the present application discloses a PCB surface defect detection method, the method comprising the following steps:

[0078] S100, constructing a training set, a validation set and a test set based on the collected PCB defect image information according to a preset proportion;

[0079] S200, preprocessing the training set to obtain a multi-defect synthetic image;

[0080] S300, constructing a target detection neural network;

[0081] S400, training the target detection neural network using the multi-defect synthetic image to obtain a defect detection model;

[0082] S500, verifying the defect detection model on the validation set to obtain a comprehensive detection model;

[0083] Testing the comprehensive detection model on the test set to obtain a PCB surface defect detection result image in the test set.

[0084] The PCB surface defect detection method disclosed in the present application, the obtained multi-defect synthetic image contains multiple types of defects, effectively increasing the sample diversity; through the constructed target detection neural network, it can fully pay attention to the connection between defects, and give feedback to the image generation network, synthesize more samples that are beneficial to improve the detection precision, increase the precision of defect detection, and ensure the authenticity of the generated image, thereby improving the generalization ability of the defect detection model, effectively alleviating the foreground-foreground imbalance problem.

[0085] Specifically referring to Figure 2 The construction method of the training set, the validation set and the test set specifically comprises:

[0086] S110, collecting a PCB finished product image;

[0087] S120, classifying the PCB finished product image based on a preset defect type and performing label processing to obtain a PCB defect image information dataset; wherein the PCB defect image information dataset comprises a plurality of single-defect PCB defect images;

[0088] S130, dividing the PCB defect image information dataset into a training set, a validation set and a test set according to a proportion of 8:1:1.

[0089] The preset defect types include short circuit, open circuit, mouse bite, missing hole, burr and false copper.

[0090] The tag processing includes generating a corresponding xml format tag file.

[0091] Referring to Figure 3 The method for obtaining a multi-defect synthetic image specifically includes the following steps:

[0092] In S210, based on the training set, a feature extraction network is used to extract features of the first foreground mask and the first background image, to obtain a first foreground mask feature image and a first background feature image.

[0093] In S220, a synthetic network is used to process the first foreground mask feature image and the first background feature image, to obtain a multi-defect synthetic image.

[0094] By using the synthetic network, the problem of single defect in the image set and the foreground-foreground imbalance problem caused by the inability to use the connection between defects are effectively solved, the sample diversity is effectively increased, and the image contains multiple defect features, which facilitates the use of information between defects.

[0095] Specifically referring to Figure 4 The first foreground mask M1 is processed in the same dimension to obtain a second foreground mask M2 with the same dimension as the first background image B1.

[0096] The second foreground mask M2 and the first background image B1 are fused by using an image processing method to obtain a first synthetic image F1.

[0097] The first background image B1, the second foreground mask M2 and the first synthetic image F1 are processed by using a spatial transformation network to obtain a second synthetic image F2.

[0098] A first defect image Q1 with the same shape as the first foreground mask M1 is extracted in the second background image B2.

[0099] The first defect image Q1 and the second synthetic image F2 are fused to obtain a third synthetic image F3.

[0100] A first background mask A1 with the same shape as the first defect image Q1 is extracted in the second background image B2.

[0101] The first background mask A1 and the third synthetic image F3 are fused to obtain a fourth synthetic image F4.

[0102] It is judged whether the fourth synthetic image F4 contains at least three defects, if yes, the fourth synthetic image is the multi-defect synthetic image.

[0103] If not, a loop step is executed.

[0104] In this embodiment, the image processing method is preferably α-fusion.

[0105] This method effectively solves the problem that too many positive samples affect the generalization ability of subsequent object detection networks.

[0106] Reference Figure 5 The loop steps include: in the m-th background image B m Extract a foreground mask M that is the same as the nth foreground mask. n Image Q of the nth defect with the same shape n Where m≥3, n=m-1;

[0107] The nth defect image Q n Synthesized image F with the sth image s By fusing, the (s+1)th synthesized image F is obtained. s+1 Where s = 2m - 2;

[0108] Background image B at the mth time m Extract a value from the nth defect image M n The nth background mask A with the same shape n ;

[0109] The nth background mask A n Combined with the (s+1)th image F s+1 Fusion is performed to obtain the (s+2)th synthesized image F. s+2 ;

[0110] Determine the synthesized image F at the (s+2)th digit. s+2 Does it contain at least three defects? If so, then the (s+2)th synthesized image F s+2 This is the composite image of the multiple defects.

[0111] If not, repeat the above steps until the (s+2)th synthesized image F. s+2 Does it contain at least three defects?

[0112] Specifically, when the fourth composite image does not contain at least three types of defects, m=3, n=2, s=4, as follows:

[0113] Extract a second defect image Q2 that has the same shape as the second foreground mask M2 from the third background image B3;

[0114] The second defect image Q2 is fused with the fourth composite image F4 to obtain the fifth composite image F5;

[0115] Extract a second background mask A2 with the same shape as the second defect image Q2 from the third background image B3;

[0116] Fuse the second background mask A2 and the fifth synthetic image F5 to obtain a sixth synthetic image F6.

[0117] Determine whether the sixth synthetic image F6 contains at least three defects, if yes, the sixth synthetic image F6 is the multi-defect synthetic image, if not, continue to cycle according to the above steps until the s+2th synthetic image F s+2 contains at least three defects.

[0118] It should be noted that the second background image and the third background image are selected to be different background images from the first background image, i.e., previously used background images, which will not be used next time, but a new background image is randomly selected.

[0119] Referring to Figure 6 In this embodiment, the target detection neural network includes convolution layers, pooling layers and fully connected layers connected in series; the fully connected layers include a related module (i.e., a Relation module) and a deletion network module, the related module is used for correlation processing between different defects of the multi-defect synthetic image, and the deletion network module is used for deleting inappropriate regions of interest to obtain a preset region of interest set (i.e., a most appropriate region of interest set).

[0120] Specifically, the Relation module uses a Self-Attention mechanism, the input of the mechanism is composed of a query and a set of key-value pairs, the output can be calculated by a weighted sum of values, and the query, the key, the value and the output are all vectors. A Softmax function is used in the calculation process to obtain the weight of the value. Given a query Query (Q in the formula), all Keys (K in the formula) and Values (V in the formula), the output value is a weighted average of the input values:

[0121]

[0122] The Relation module needs to calculate the relationship of the described objects. An object is composed of geometric features f G and appearance features f A . Here, f G is a 4-dimensional object bounding box, and f A is a key feature for the Relation module. Given an input set of N objects The entire object relationship feature f R (n) set for the nth object is calculated as:

[0123]

[0124] The output is a weighted sum of appearance features from other objects, which is calculated by WV Linear transformation (corresponding to the value V in Self-Attention). Relationship weight ω mn This indicates the influence of other objects. The calculation method is as follows:

[0125]

[0126] It is calculated using the dot product method, similar to the Self-Attention formula.

[0127]

[0128] W K and W Q Both are matrices, and their function is similar to K and Q in the Self-Attention formula. They convert the original features... and The features are projected into a subspace to measure their matching degree. The dimension of the projected features is d. k .

[0129] The geometric weights are calculated as follows:

[0130]

[0131] First, the geometric features of the two objects are embedded into a high-dimensional representation, denoted as ε. G To ensure its invariance during translation and scaling, a 4D relative geometric feature is used, such as...

[0132]

[0133] This 4-dimensional feature is embedded into a high-dimensional representation using a self-attention mechanism, which calculates cosine and sine functions for different wavelengths. The embedded feature dimension is d. g Then, use W G A linear mapping is performed on this feature vector to obtain a scalar, which is then truncated to 0, thus achieving a non-linear activation effect similar to ReLU.

[0134] The Relation module aggregates all Nr relation features and enhances the appearance features of the input object through addition.

[0135]

[0136] Here, `Concat[]` is used to aggregate multiple relation features. To match the channel size, each... The output channel is set as the input feature. Size

[0137] There are three steps for the deletion network. First, the 1024-dimensional features are fused with the classification scores to generate appearance features. Second, the relation module transforms these appearance features for all objects. Finally, the transformed features for each object are passed through a linear classifier and Sigmoid to output a probability ∈ [0, 1]. The Relation module is the core of the network. It can use information from multiple sources (bounding boxes, raw appearance features, and classification scores) for effective end-to-end learning.

[0138] To make the deletion more effective, the scores need to be converted into rankings, rather than just using its value, i.e., the N input objects are sorted in descending order of scores. Each object is assigned a corresponding ranking ∈ [1, N] by computing. The scalar ranking is then embedded into a higher-dimensional 128-dimensional feature. Both the ranking feature and the original 1024-dimensional appearance feature are transformed into 128-dimensional, and then the two are fused as the input of the relation module.

[0139] Finally, by setting a predefined threshold η, the detection box with the highest score whose IoU with the ground truth box is ≥ η is correct, and the others are redundant.

[0140] Through the target detection neural network constructed by the application, the appearance and geometric information between defects are utilized to alleviate the foreground-foreground imbalance problem, and the deletion network module is further added to select the optimal solution in the regression process, thereby further improving the detection speed.

[0141] Reference Figure 7 Further, in other embodiments of the application, the application discloses a PCB surface defect detection method, which specifically comprises the following steps:

[0142] Based on the collected original PCB defect image information, a training set, a verification set and a test set are constructed according to a preset proportion;

[0143] The training set is preprocessed to obtain a multi-defect synthetic image;

[0144] The multi-defect synthetic image is subjected to secondary discrimination to obtain a real synthetic image;

[0145] A target detection neural network is constructed;

[0146] The target detection neural network is trained using the real synthetic image to obtain a defect detection model;

[0147] The defect detection model is verified on the verification set;

[0148] The defect detection model is tested on the test set to obtain a PCB surface defect detection result image.

[0149] Or, the defect detection model is verified on the verification set to obtain a comprehensive detection model; the comprehensive detection model is tested on the test set to obtain a PCB surface defect detection result image in the test set.

[0150] With reference to Figure 8 The real synthetic image is obtained by the following steps:

[0151] The discriminator is used to perform secondary discrimination on the multi-defect synthetic image.

[0152] It is determined that the multi-defect synthetic image is a real synthetic image if the output true probability value is greater than the preset value.

[0153] If not, it is determined that the multi-defect synthetic image is a false synthetic image, and the synthetic network changes the training parameters to re-generate the multi-defect synthetic image.

[0154] The training parameters include the learning rate and the loss function.

[0155] In this embodiment, the discriminator used is a trained discriminator, and the specific training process is as follows: the collected PCB defect image information (i.e., single-defect image information) and multi-defect synthetic images are both input to the discriminator, and the generator and the discriminator are trained in an adversarial manner using a generative adversarial network; if the discriminator can determine that the multi-defect synthetic image is false, it indicates that the performance of the generator is not good enough, and the generated image is too fake, thereby improving the generation performance of the generator; until the multi-defect synthetic image output by the generator with improved performance is determined to be real by the discriminator, it indicates that the generator has good performance, and the obtained multi-defect synthetic image is also optimal.

[0156] In this embodiment, the multi-defect synthetic image is obtained through the synthetic network, and therefore, during the adversarial training process, the performance of the generator is directly synchronized with the synthetic network. When the multi-defect synthetic image is determined to be a false image, it is fed back to the synthetic network in real time, triggering the synthetic network to change the training parameters and re-generate the multi-defect synthetic image. Through this adversarial form, a high-performance synthetic network is obtained, ensuring that the obtained multi-defect synthetic image is more real and preventing overfitting of the target detection network.

[0157] For PCB image defect detection, the evaluation indicators mainly used are AP and mAP. AP is the average precision of each class, and mAP is the average precision of all classes.

[0158] Table 1 shows the comparison test results of replacing the target detection network with the following target detection networks under the premise of using image synthesis and discriminator.

[0159] As can be seen from Table 1, the mAP obtained by the method herein is 96.37%, which is improved by 7.8% compared with Faster R-CNN, improved by 6.31% compared with Mask R-CNN, and then improved by 5.65% compared with the one-stage SSD. In terms of PCB defect detection, the method proposed herein achieves good results; the detection effect of different types of defects is greatly improved, and the detection gap of different types of defects is greatly reduced.

[0160] Table 1: Comparison test of PCB defect detection

[0161]

[0162] Reference Figure 9 The second aspect of the application discloses a PCB surface defect detection system, comprising:

[0163] A division module configured to construct a training set, a validation set and a test set according to a preset proportion based on the collected PCB defect image information;

[0164] A preprocessing module configured to preprocess the training set to obtain a multi-defect synthetic image;

[0165] A construction module configured to construct a target detection neural network;

[0166] A training module configured to train the target detection neural network using the multi-defect synthetic image to obtain a defect detection model;

[0167] A verification module configured to verify the defect detection model on the validation set to obtain a comprehensive detection model;

[0168] A test module configured to test the comprehensive detection model on the test set to obtain a PCB surface defect detection result image in the test set.

[0169] The electronic device according to the embodiments of the present disclosure includes a memory and a processor. The memory is configured to store non-transitory computer readable instructions. Specifically, the memory can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0170] The processor can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction executing capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the PCB surface defect detection method according to the embodiments of the present disclosure.

[0171] Those skilled in the art will understand that, in order to solve the technical problem of how to obtain a good user experience effect, the present embodiment can also include well-known structures such as a communication bus, an interface, and the like, which should also be included in the protection scope of the present disclosure.

[0172] As Figure 10 A structural schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. The structural schematic diagram shows a structure suitable for implementing the electronic device according to the embodiments of the present disclosure. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0173] As Figure 10 As shown, the electronic device can include a processing device (such as a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) or loaded from a storage device into a random access memory (RAM). Various programs and data required for operation of the electronic device are also stored in the RAM. The processing device, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0174] Generally, the following devices can be connected to the I / O interface: input devices including, for example, sensors or visual information acquisition devices; output devices including, for example, display screens; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication devices can allow the electronic device to communicate with other devices (such as edge computing devices) to exchange data wirelessly or through wires. Although Figure 10 The electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices can be alternatively implemented or provided.

[0175] In particular, according to embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the PCB surface defect detection method of the embodiments of the present disclosure are performed.

[0176] Detailed descriptions of the embodiments of the present disclosure can refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0177] The computer-readable storage medium according to the embodiments of the present disclosure has non-transitory computer-readable instructions stored thereon. When the non-transitory computer-readable instructions are run by a processor, all or part of the steps of the PCB surface defect detection method of the embodiments of the present disclosure described above are performed.

[0178] The computer-readable storage medium described above includes, but is not limited to, optical storage media (such as CD-ROM and DVD), magneto-optical storage media (such as MO), magnetic storage media (such as magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (such as memory card), and media with built-in ROM (such as ROM cartridge).

[0179] Detailed descriptions of the embodiments of the present disclosure can refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0180] The basic principles of the present disclosure are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above-mentioned details do not limit the present disclosure to be necessarily implemented with the above-mentioned specific details.

[0181] In this disclosure, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The block diagram of the devices, apparatus, equipment, systems referred to in this disclosure is merely illustrative and not intended to imply the necessity or arrangement of the connections, arrangement, configuration as shown in the block diagram. As will be appreciated by those skilled in the art, the devices, apparatus, equipment, systems can be connected, arranged, configured in any manner. The words comprising, including, having and the like are to be open ended. As used in this document, the conjunction "or" is to be interpreted in the inclusive sense, i.e. as meaning one or the other, or both. As used in this document, the words "and" and "or" are to be interpreted as having the meaning indicated in the phrase "and / or". As used in this document, the word "such as" is to be interpreted as meaning "such as, but not limited to". As used in this document, the word "for example" is to be interpreted as meaning "by way of example, not by way of limitation".

[0182] Also, as used in this document, the word "or" in the list of items prefaced by "at least one of indicates a disjunctive sense such that, for example, "at least one of A, B or C" means A or B or C or AB or AC or BC or ABC (i.e. A and B and C). Furthermore, the phrase "example of" does not mean an example that is preferred or better than other examples.

[0183] It is also important to note that the systems and methods of the present disclosure can be embodied in a variety of forms including, but not limited to, a data processor, a computer program product, a computer, one or more tangible computer readable storage devices, one or more computer memories, one or more programmable logic devices, one or more application specific devices, one or more computers, one or more processors, one or more microprocessors, one or more microcomputers, one or more microcontrollers, one or more microcontrollers, one or more microprocessors, one or more state machines, one or more integrated circuits, one or more other components, or any combination thereof, and that the systems and methods can comprise, consist of, or consist essentially of such forms.

[0184] Various changes, modifications and alterations in the teachings and techniques described herein can be made without departing from the teachings and techniques defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not intended to be limited to the process, machine, manufacture, composition of matter, means, methods, and steps described in the specification. Any process, machine, manufacture, composition of matter, means, methods, and steps present in the practice of the disclosure can be utilized in the practice of the present disclosure, even though the process, machine, manufacture, composition of matter, means, methods, and steps are not expressly described in the specification. Accordingly, the appended claims include within their scope the process, machine, manufacture, composition of matter, means, methods, and steps present in the practice of the present disclosure.

[0185] The above description of the disclosed aspects is intended to be illustrative and not restrictive. Many modifications and variations of the aspects described herein will be apparent to those skilled in the art from the teachings that follow without departing from the scope of the disclosure. The general scope of the disclosure is defined by the appended claims and not by the specific illustrative examples that are provided herein. Accordingly, the disclosure is intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims. The claims should not be limited to the various aspects and features described herein, but should be given their broadest interpretation consistent with the patent statutes.

[0186] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A method for detecting defects on the surface of a PCB, characterized in that, Includes the following steps: Based on the collected raw PCB defect image information, a training set, a validation set, and a test set are constructed according to a preset ratio; The training set is preprocessed to obtain multi-defect synthetic images; Construct an object detection neural network; The target detection neural network is trained using the multi-defect synthetic image to obtain a defect detection model; The defect detection model is validated on the validation set to obtain a comprehensive detection model; The comprehensive detection model is tested on the test set to obtain the PCB surface defect detection result map in the test set; The step of preprocessing the training set to obtain multi-defect synthetic images includes: Based on the training set, a feature extraction network is used to extract features from the first foreground mask and the first background image to obtain the first foreground mask feature image and the first background feature image. The first foreground mask feature image and the first background feature image are processed using a synthetic network to obtain the multi-defect synthetic image; The step of processing the first foreground mask feature image and the first background feature image using a synthetic network to obtain the multi-defect synthetic image includes: The first foreground mask feature image is processed in the same dimension to obtain a second foreground mask with the same dimension as the first background feature image. The second foreground mask and the first background feature image are fused using image processing methods to obtain a first composite image; A spatial transformation network is used to process the first background feature image, the second foreground mask, and the first composite image to obtain the second composite image; Extract a first defect image from the second background image that has the same shape as the first foreground mask feature image; The first defect image is fused with the second synthesized image to obtain a third synthesized image; Extract a first background mask from the second background image that has the same shape as the first defect image; The first background mask is fused with the third composite image to obtain a fourth composite image; Determine whether the fourth composite image contains at least three types of defects; if so, the fourth composite image is the multi-defect composite image. If not, execute the loop step.

2. The PCB surface defect detection method according to claim 1, characterized in that, The training set, validation set, and test set are constructed based on the acquired PCB defect image information according to a preset ratio, including: Acquire images of finished PCB products; The PCB finished product images are classified and tagged based on preset defect types to obtain a PCB defect image information dataset. The PCB defect image information dataset is divided into the training set, the validation set, and the test set in a ratio of 8:1:

1.

3. The PCB surface defect detection method according to claim 1, characterized in that, The cyclic steps include: Extract a defect image of the nth degree with the same shape as the nth foreground mask from the mth background image; where m≥3, n=m-1; The nth defect image is fused with the sth composite image to obtain the (s+1)th composite image; where s = 2m-2; Extract a background mask of the nth shape that is identical to the defect image from the mth background image; The nth background mask is fused with the (s+1)th composite image to obtain the (s+2)th composite image; Determine whether the (s+2)th synthesized image contains at least three types of defects. If so, the (s+2)th synthesized image is the multi-defect synthesized image. If not, repeat the above steps until the (s+2)th synthesized image contains at least three defects.

4. The PCB surface defect detection method according to claim 3, characterized in that, It also includes the following steps: A discriminator is used to perform secondary discrimination on the multi-defect composite image; Determine whether the output true probability value is greater than a preset value. If so, determine that the multi-defect synthesized image is a real synthesized image. If not, the multi-defect synthesized image is determined to be a fake synthesized image, triggering the synthesis network to change its training parameters and regenerate the multi-defect synthesized image; The training parameters include the learning rate and the loss function.

5. The PCB surface defect detection method according to claim 1, characterized in that, The target detection neural network includes convolutional layers, pooling layers, and fully connected layers connected in series. The fully connected layer includes a correlation module and a deletion network module. The correlation module is used to perform correlation processing between different defects in the multi-defect synthetic image. The network deletion module is used to delete unsuitable regions of interest and obtain a preset set of regions of interest.

6. A PCB surface defect detection system, characterized in that, include: The module is divided and configured to construct training, validation, and test sets based on the collected PCB defect image information according to a preset ratio; The preprocessing module is configured to preprocess the training set to obtain a multi-defect synthetic image; The module is configured to build an object detection neural network; The training module is configured to train the target detection neural network using the multi-defect synthetic image to obtain a defect detection model; The verification module is configured to verify the defect detection model on the verification set; The testing module is configured to test the defect detection model on the test set and obtain a PCB surface defect detection result image. The step of preprocessing the training set to obtain multi-defect synthetic images includes: Based on the training set, a feature extraction network is used to extract features from the first foreground mask and the first background image to obtain the first foreground mask feature image and the first background feature image. The first foreground mask feature image and the first background feature image are processed using a synthetic network to obtain the multi-defect synthetic image; The step of processing the first foreground mask feature image and the first background feature image using a synthetic network to obtain the multi-defect synthetic image includes: The first foreground mask feature image is processed in the same dimension to obtain a second foreground mask with the same dimension as the first background feature image. The second foreground mask and the first background feature image are fused using image processing methods to obtain a first composite image; A spatial transformation network is used to process the first background feature image, the second foreground mask, and the first composite image to obtain the second composite image; Extract a first defect image from the second background image that has the same shape as the first foreground mask feature image; The first defect image is fused with the second synthesized image to obtain a third synthesized image; Extract a first background mask from the second background image that has the same shape as the first defect image; The first background mask is fused with the third composite image to obtain a fourth composite image; Determine whether the fourth composite image contains at least three types of defects; if so, the fourth composite image is the multi-defect composite image. If not, execute the loop step.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the PCB surface defect detection method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the PCB surface defect detection method according to any one of claims 1-5.

Images