Defect detection method, device and equipment for printed circuit board and storage medium

Abstract

The invention relates to the field of defect detection, in particular to a defect detection method, device and equipment for a printed circuit board and a storage medium, which are used for carrying out feature fusion upsampling and convolution connection feature fusion on a multi-scale backbone feature map of a constructed printed circuit board image, so that the calculation burden is reduced, and the detection efficiency is improved. According to the method, the accuracy and efficiency of feature extraction of the small target are improved, the method is used for defect detection of the printed circuit board, comprehensive and sufficient defect detection of the printed circuit board image is achieved, and the accuracy and efficiency of defect detection of the printed circuit board are improved.

Description

Technical Field

[0001] The present invention relates to the field of defect detection, and in particular to a defect detection method, device, apparatus and storage medium for a printed circuit board. Background Art

[0002] Due to process and other issues, there are manufacturing defects in the production of printed circuit boards (PCBs). Common PCB defects mainly include hole defects, line defects, pad defects, solder mask defects and other problems. These defects will cause unstable circuit board connections and signal transmission interference between boards, which will lead to a shortened service life of assembled electronic products, malfunctions, unstable operation of electronic products and other problems. Therefore, strict quality control is required at all stages of production through a variety of testing methods.

[0003] At present, the traditional printed circuit board defect detection mainly includes manual inspection (visual inspection), automatic optical inspection, X-ray inspection and other detection methods. However, traditional detection has the characteristics of low efficiency, long time consumption and high missed detection rate. With the continuous advancement of computer vision technology and the continuous upgrading of hardware resources (such as GPU), PCB defect detection algorithms based on computer vision technology are widely used in PCB defect detection. Although they can solve the problem of detection accuracy, the detection speed is slow and cannot meet the rapid detection requirements of a large number of PCB boards passing through the assembly line in factory production scenarios. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a defect detection method, device, equipment and storage medium for a printed circuit board, perform feature fusion upsampling and convolution connection feature fusion on the multi-scale backbone feature map of the constructed printed circuit board image, reduce the computational burden, improve the accuracy and efficiency of feature extraction of small targets, and use them to perform defect detection on the printed circuit board, thereby achieving comprehensive and sufficient defect detection of the printed circuit board image, and improving the accuracy and efficiency of defect detection on the printed circuit board.

[0005] In a first aspect, an embodiment of the present application provides a method for defect detection of a printed circuit board, comprising the following steps:

[0006] Obtaining a printed circuit board image to be inspected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module;

[0007] Inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales;

[0008] Inputting backbone feature maps of several scales into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain feature fusion upsampling maps;

[0009] The feature fusion up-sampling graphs are respectively input into the convolutional connection feature fusion module for convolutional connection feature fusion to obtain convolutional connection feature fusion graphs of several scales;

[0010] The convolutional connection feature fusion graphs of several scales are input into the detection network for target detection to obtain defect detection results of the printed circuit board image to be detected.

[0011] In a second aspect, an embodiment of the present application provides a defect detection device for a printed circuit board, comprising:

[0012] A data acquisition module, used to obtain a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module;

[0013] A feature extraction module, used for inputting the printed circuit board image to be detected into the backbone network for feature extraction, and obtaining backbone feature maps of several scales;

[0014] A feature sampling module, used for inputting backbone feature maps of several scales into a feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map;

[0015] A feature fusion module, used for inputting the feature fusion up-sampling graphs into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion graphs of several scales;

[0016] The defect detection module is used to input the convolution connection feature fusion graphs of several scales into the detection network for target detection to obtain the defect detection result of the printed circuit board image to be detected.

[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the printed circuit board defect detection method as described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the printed circuit board defect detection method as described in the first aspect are implemented.

[0019] In an embodiment of the present application, a method, apparatus, device and storage medium for defect detection of a printed circuit board are provided, and feature fusion upsampling and convolution connection feature fusion are performed on a multi-scale backbone feature map of a constructed printed circuit board image, thereby reducing the computational burden and improving the accuracy and efficiency of feature extraction of small targets, so as to perform defect detection of the printed circuit board, realize comprehensive and sufficient defect detection of the printed circuit board image, and improve the accuracy and efficiency of defect detection of the printed circuit board.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a process flow of a printed circuit board defect detection method provided by an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the process of S2 in a defect detection method for a printed circuit board provided in one embodiment of the present application;

[0023] Figure 3 A schematic diagram of the process of S3 in a defect detection method for a printed circuit board provided in one embodiment of the present application;

[0024] Figure 4 A schematic diagram of the process of S4 in a defect detection method for a printed circuit board provided in one embodiment of the present application;

[0025] Figure 5 A schematic diagram of the process of S42 in a defect detection method for a printed circuit board provided in one embodiment of the present application;

[0026] Figure 6 A schematic diagram of the process of S5 in a defect detection method for a printed circuit board provided in one embodiment of the present application;

[0027] Figure 7 A schematic diagram of the process of S6 in a printed circuit board defect detection method provided by another embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a defect detection device for a printed circuit board provided in one embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0033] See also Figure 1 , Figure 1 A schematic flow chart of a defect detection method for a printed circuit board provided in one embodiment of the present application, the method comprising the following steps:

[0034] S1: Obtaining a printed circuit board image to be inspected and a preset defect detection model.

[0035] The execution subject of the printed circuit board defect detection method is a detection device of the printed circuit board defect detection method (hereinafter referred to as the detection device). In an optional embodiment, the detection device can be a computer device, a server, or a server cluster composed of multiple computer devices.

[0036] In this embodiment, the detection device may obtain the printed circuit board image to be detected input by the user, or may obtain the printed circuit board image to be detected from a preset database.

[0037] The detection equipment obtains a preset defect detection model, wherein the photovoltaic panel defect detection model is an improved Yolov5 model, which is a target detection model that can solve various problems in the actual operation of defect detection of printed circuit boards and takes into account both computational complexity and accuracy, including a backbone network (Backbone), a neck network (Neck) and a detection network (YoloHead), wherein the neck network includes a feature fusion upsampling module and a convolutional connection feature fusion module.

[0038] S2: Inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales.

[0039] In this embodiment, the detection device inputs the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales.

[0040] The backbone network includes a downsampling module, a convolution fusion module, and a spatial pyramid pooling module connected in sequence; the convolution fusion module includes a plurality of convolution fusion submodules connected in sequence, and the convolution fusion submodule includes a convolution unit and a depth-separable convolution unit connected in sequence; please refer to Figure 2 , Figure 2 The schematic flow diagram of S2 in the defect detection method of a printed circuit board provided in one embodiment of the present application includes steps S21 to S23, which are specifically as follows:

[0041] S21: down-sampling the printed circuit board image to be detected through the down-sampling module in turn to obtain a feature extraction image.

[0042] The downsampling module adopts the focus module, which is a convolutional neural network layer for feature extraction, and is used to compress and combine the information in the input feature map to extract a higher level feature representation.

[0043] In this embodiment, the detection device sequentially downsamples the printed circuit board image to be detected through the downsampling module to obtain a feature extraction map, so as to reduce the amount of calculation and the amount of parameters.

[0044] S22: Using the feature extraction map as the first input feature map of the first convolution fusion sub-module of the convolution fusion module, and processing it in sequence through the convolution unit and the depth-separable convolution unit to obtain the first output feature map of the first convolution fusion sub-module, and using the first output feature map as the first input feature map of the next convolution fusion sub-module, and repeating the process until the first output feature map of the last convolution fusion sub-module is obtained.

[0045] The convolution unit is a convolution layer, and the depth-wise separable convolution unit is a unit composed of three C3 (cspbottleneck with3conv) blocks.

[0046] In this embodiment, the detection device uses the feature extraction map as the first input feature map of the first convolution fusion sub-module of the convolution fusion module, and processes it in sequence through the convolution unit and the depth-separable convolution unit to obtain the first output feature map of the first convolution fusion sub-module, and uses the first output feature map as the first input feature map of the next convolution fusion sub-module, and repeats the process until the first output feature map of the last convolution fusion sub-module is obtained.

[0047] S23: The first output feature map of the last convolution fusion submodule is pooled through the spatial pyramid pooling module to obtain a pooled processing map as the backbone feature map of the last scale, and the first output feature maps of other convolution fusion submodules are used as the backbone feature maps of the corresponding scales to obtain backbone feature maps of several scales.

[0048] In this embodiment, the detection device performs pooling processing on the first output feature map of the last convolution fusion sub-module through the spatial pyramid pooling module to obtain a pooling processing map as the backbone feature map of the last scale, and uses the first output feature maps of other convolution fusion sub-modules as the backbone feature maps of the corresponding scales to obtain backbone feature maps of several scales.

[0049] Specifically, the detection device convolves the first output feature map of the last convolution fusion submodule, extracts feature information in the horizontal and vertical directions, and generates an attention feature map through convolution transformation; finally, the attention feature map is multiplied with the first output feature map of the last convolution fusion submodule to obtain a pooling processing map as the backbone feature map of the last scale, so as to fully extract the detail information of the printed circuit board image to be detected and improve the accuracy of defect detection.

[0050] S3: Inputting backbone feature maps of several scales into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map.

[0051] In this embodiment, the detection device inputs backbone feature maps of several scales into the feature fusion upsampling module in the neck network to perform feature fusion upsampling to obtain a feature fusion upsampling map.

[0052] The feature fusion upsampling module includes a plurality of sequentially connected feature fusion upsampling submodules, wherein the feature fusion upsampling submodule includes a sequentially connected convolution unit, a DySample upsampling unit, a connection unit, a feature modeling unit, and a CBAM (Convolutional Block Attention Module) attention mechanism unit; please refer to Figure 3 , Figure 3 The schematic flow diagram of S3 in the defect detection method of a printed circuit board provided in one embodiment of the present application includes steps S31 to S33, which are specifically as follows:

[0053] S31: Use the backbone feature map of the last scale as the second input feature map of the first feature fusion upsampling submodule, and process the second input feature map through the convolution unit and the DySample upsampling unit in sequence to obtain the second input feature map after upsampling.

[0054] The Dysample module optimizes the generation of offsets in upsampling. It creates a sampling set through a sampling generator and uses it for dynamic upsampling, thereby focusing more on defect information in the image and reducing the impact of noise.

[0055] In this embodiment, the detection device uses the backbone feature map of the last scale as the second input feature map of the first feature fusion upsampling submodule, processes the second input feature map through the convolution unit and the DySample upsampling unit in sequence, and obtains the second input feature map after upsampling.

[0056] S32: The second input feature map after the upsampling process and the backbone feature map of the previous scale of the second input feature map are feature-connected through a connection unit to obtain a first feature connection map; the first feature connection map is subjected to attention extraction through the feature modeling unit and the CBAM attention mechanism unit to obtain a first attention feature map as the second output feature map of the first feature fusion upsampling submodule.

[0057] In this embodiment, the detection device performs feature connection on the second input feature map after the upsampling process and the backbone feature map of the previous scale of the second input feature map through a connection unit to obtain a first feature connection map.

[0058] The feature modeling unit is a unit composed of three C3 (cspbottleneck with 3conv) blocks. The feature modeling unit of the last feature fusion upsampling module adopts a linear time series modeling unit (Mamba). The main advantage of the linear time series modeling unit lies in its excellent performance on long sequence tasks and low computational complexity. It improves computational efficiency through discretization and convolution, and improves the model's ability to capture long-distance information.

[0059] The CBAM attention mechanism unit is a lightweight and effective attention mechanism that enables the model to capture and utilize key information more accurately by introducing channel attention and spatial attention.

[0060] The detection device extracts attention from the first feature connection graph through the feature modeling unit and the CBAM attention mechanism unit to obtain a first attention feature graph as the second output feature graph of the first feature fusion upsampling submodule, introduces the attention mechanism, extracts attention from the data after feature modeling, and fully extracts the feature information in the printed circuit board image.

[0061] S33: Using the second output feature map of the first feature fusion upsampling submodule as the second input feature map of the next feature fusion upsampling submodule, repeating the feature fusion upsampling until the second output feature map of the last feature fusion upsampling submodule is obtained, and using the second output feature map of the last feature fusion upsampling submodule as the feature fusion upsampling map to obtain the feature fusion upsampling map.

[0062] In this embodiment, the detection device uses the second output feature map of the first feature fusion upsampling submodule as the second input feature map of the next feature fusion upsampling submodule, and repeats the feature fusion upsampling until the second output feature map of the last feature fusion upsampling submodule is obtained, and uses the second output feature map of the last feature fusion upsampling submodule as the feature fusion upsampling map to obtain the feature fusion upsampling map.

[0063] S4: Inputting the feature fusion up-sampling graphs into the convolutional connection feature fusion modules respectively for convolutional connection feature fusion to obtain convolutional connection feature fusion graphs of several scales.

[0064] In this embodiment, the detection device inputs the feature fusion up-sampling graphs into the convolutional connection feature fusion module respectively to perform convolutional connection feature fusion, and obtains convolutional connection feature fusion graphs of several scales.

[0065] The convolutional connection feature fusion module includes a plurality of sequentially connected convolutional connection feature fusion submodules, wherein the convolutional connection feature fusion submodule includes sequentially connected convolution units, connection units, linear time series modeling units and CBAM attention mechanism units; please refer to Figure 4 , Figure 4 The schematic flow diagram of S4 in the defect detection method for a printed circuit board provided in one embodiment of the present application includes steps S41 to S43, which are specifically as follows:

[0066] S41: Using the feature fusion upsampling map as the third input feature map of the first convolution connection feature fusion submodule, and performing convolution processing on the third input feature map through a convolution unit to obtain a third input feature map after the convolution processing; using a backbone feature map of a scale above the third input feature map as a to-be-connected map, and performing feature connection on the third input feature map after the convolution processing and the to-be-connected map through a connection unit to obtain a second feature connection map.

[0067] In this embodiment, the detection device uses the feature fusion upsampling map as the third input feature map of the first convolution connection feature fusion submodule, and performs convolution processing on the third input feature map through a convolution unit to obtain a third input feature map after convolution processing.

[0068] The detection device uses a backbone feature map of a scale on the third input feature map as a to-be-connected map, performs feature connection on the third input feature map after convolution processing and the to-be-connected map through a connection unit, and obtains a second feature connection map.

[0069] S42: performing feature modeling on the second feature connection graph through the linear time series modeling unit to obtain a linear time series modeling graph; performing attention extraction on the linear time series modeling graph through the CBAM attention mechanism unit to obtain a second attention feature graph as the third output feature graph of the first convolution connection feature fusion submodule;

[0070] In this embodiment, the detection device performs feature modeling on the second feature connection graph through the linear time series modeling unit to obtain a linear time series modeling graph. The linear time series modeling method of selecting state space is adopted to perform feature modeling on the second feature connection graph to construct a long-range dependency relationship and maintain the linear complexity of the feature graph, thereby reducing the computational burden and improving the accuracy and efficiency of feature extraction of small targets.

[0071] The detection device extracts attention from the linear time series modeling graph through the CBAM attention mechanism unit to obtain a second attention feature graph as the third output feature graph of the first convolution connection feature fusion submodule, introduces the attention mechanism, extracts attention from the data after feature modeling, fully extracts the feature information in the printed circuit board image, and improves the accuracy and efficiency of defect detection of the printed circuit board.

[0072] The linear time series modeling unit includes a linear projection unit, a convolution unit and a state space unit; see Figure 5 , Figure 5 The schematic flow diagram of S42 in the defect detection method of a printed circuit board provided in one embodiment of the present application includes steps S421 to S423, which are specifically as follows:

[0073] S421: Input the second feature connection map into the linear projection unit for feature mapping to obtain a feature mapping map, input the feature mapping map into the convolution unit for convolution to obtain an intermediate convolution feature map; perform nonlinear processing on the intermediate convolution feature map to obtain the intermediate convolution feature map after nonlinear processing.

[0074] In order to project the feature tensor to a higher dimension for processing and to capture more detailed features, in this embodiment, the detection device inputs the second feature connection map into the linear projection unit for feature mapping to obtain a feature mapping map.

[0075] The detection device inputs the feature map into the convolution unit for convolution to obtain an intermediate convolution feature map; performs nonlinear processing on the intermediate convolution feature map to obtain the intermediate convolution feature map after nonlinear processing, so as to capture the information of neighboring tokens before inputting the feature map into the state space unit, thereby improving the accuracy of feature modeling.

[0076] S422: Input the intermediate convolution feature map after the nonlinear processing as the fourth input feature map of the state space unit, and obtain the fourth output feature map of the state space unit as the state space feature map according to a preset state space calculation algorithm.

[0077] The state space calculation algorithm is:

[0078] h k =Ah k-1 +Bx k

[0079] y k =Ch k

[0080] In the formula, h kis the hidden layer state diagram for the next time step, h k-1 is the hidden layer state diagram of the current time step, x k is the fourth input feature map for the next time step, y k is the fourth output feature map of the next time step, A, B, and C are the state transfer matrix, input matrix, and output matrix, respectively, and the state transfer matrix, input matrix, and output matrix are all matrices updated by back propagation during the training process of the linear time series modeling unit.

[0081] In this embodiment, the detection device inputs the intermediate convolution feature map after the nonlinear processing as the fourth input feature map of the state space unit, and obtains the fourth output feature map of the state space unit as the state space feature map according to a preset state space calculation algorithm.

[0082] S423: Perform nonlinear processing on the state-space feature graph and the feature map to obtain the state-space feature graph and the feature map after nonlinear processing, perform residual connection on the state-space feature graph and the feature map after nonlinear processing to obtain a residual connection feature map, input the residual connection feature map into the linear projection unit for feature mapping, and obtain a linear time series modeling map.

[0083] In this embodiment, the detection device performs nonlinear processing on the state-space feature graph and the feature map to obtain the state-space feature graph and the feature map after nonlinear processing, performs residual connection on the state-space feature graph and the feature map after nonlinear processing to obtain a residual connection feature map, and inputs the residual connection feature map into the linear projection unit for feature mapping to obtain a linear time series modeling map.

[0084] S43: Using the third output feature map of the first convolutional connection feature fusion submodule as the third input feature map of the next convolutional connection feature fusion submodule and the map to be connected, repeating the convolutional connection feature fusion until the linear time series modeling map output by the linear time series modeling unit of the last convolutional connection feature fusion submodule is obtained, and using the linear time series modeling map as the convolutional connection feature fusion map to obtain convolutional connection feature fusion maps of several scales.

[0085] In this embodiment, the detection device uses the third output feature map of the first convolutional connection feature fusion submodule as the third input feature map of the next convolutional connection feature fusion submodule and the map to be connected, and repeats the convolutional connection feature fusion until the linear time series modeling map output by the linear time series modeling unit of the last convolutional connection feature fusion submodule is obtained, and uses the linear time series modeling map as the convolutional connection feature fusion map to obtain convolutional connection feature fusion maps of several scales.

[0086] S5: Inputting the convolutional connection feature fusion graphs of several scales into the detection network for target detection to obtain defect detection results of the printed circuit board image to be detected.

[0087] In this embodiment, the detection device inputs the convolutional connection feature fusion graphs of several scales into the detection network for target detection to obtain defect detection results of the printed circuit board image to be detected.

[0088] See also Figure 6 , Figure 6 The flow diagram of S5 in the defect detection method of a printed circuit board provided in one embodiment of the present application includes step S51, which is specifically as follows:

[0089] S51: According to the convolution connection feature fusion map and the preset detector, several predicted areas of the printed circuit board image to be detected and the coordinate information and defect category information of the several predicted areas are obtained as the defect detection result of the printed circuit board image to be detected.

[0090] In this embodiment, the detection device obtains several predicted areas of the printed circuit board image to be detected and coordinate information and defect category information of the several predicted areas according to the convolutional connection feature fusion map and the preset detector as the defect detection result of the printed circuit board image to be detected, wherein the coordinate information includes the center point coordinate parameters, width parameters and height parameters of the predicted area, which are used to indicate the position and size of the predicted area, and the defect category information is used to indicate the defect category of the predicted area.

[0091] A linear time series modeling method with selected state space is adopted to perform feature fusion upsampling and convolution connection feature fusion on the multi-scale backbone feature map of the constructed printed circuit board image, which reduces the computational burden and improves the accuracy and efficiency of feature extraction of small targets for printed circuit board defect detection. It realizes comprehensive and sufficient defect detection of printed circuit board images and improves the accuracy and efficiency of printed circuit board defect detection.

[0092] In an optional embodiment, the method further comprises step S7: training the defect detection model. Figure 7 , Figure 7 The flowchart of S6 in the defect detection method of a printed circuit board provided in another embodiment of the present application includes steps S61 to S62, which are specifically as follows:

[0093] S61: Obtain a plurality of sample printed circuit board images and a plurality of defect label results of the sample printed circuit board images.

[0094] In this embodiment, the detection device obtains a number of sample printed circuit board images and a number of defect label results of the sample printed circuit board images, wherein the defect label results include coordinate information of a number of label areas and defect category information.

[0095] Specifically, the detection equipment uses the public data set PKU-Market-PCB of the Artificial Intelligence Laboratory of Peking University. The detection equipment enhances the data set by scaling, brightness transformation, salt and pepper noise, Gaussian noise, translation and rotation of the images in the PKU-Market-PCB data set to obtain several sample printed circuit board images.

[0096] S62: Inputting a number of the sample printed circuit board images into the defect detection model to be trained to obtain defect detection results of the sample printed circuit board images; obtaining a regression loss value according to the defect detection results of the sample printed circuit board images, the defect label results and the preset Shape-IoU loss function, and training the defect detection model to be trained according to the regression loss value.

[0097] The Shape-IoU loss function is:

[0098]

[0099] L Shape-IoU =1-IoU+distance shape +0.5×Ω shape

[0100] Where, distance shape is the point distance, scale is the scale factor, h gt is the height parameter of the label area, w gt is the width parameter of the label area, x c is the horizontal coordinate parameter in the center point coordinate parameter of the prediction area, is the horizontal coordinate parameter in the center coordinate parameter of the label area, and c represents (x c ,y c )as well as The hypotenuse length of the minimum area between is the ordinate parameter in the center coordinate parameter of the label area, Ω shape is the shape similarity loss value, wt is the weight of the shape similarity loss, Θ is the shape similarity loss index, L Shape-IoU is the regression loss value, IoU is the intersection over union ratio, which is the ratio of the intersection and union of the prediction area and the label area.

[0101] In this embodiment, the detection device inputs several of the sample printed circuit board images into the defect detection model to be trained to obtain defect detection results of the several sample printed circuit board images; obtains a regression loss value based on the defect detection results of the several sample printed circuit board images, the defect label results and the preset Shape-IoU loss function, and trains the defect detection model to be trained based on the regression loss value, thereby improving the accuracy and efficiency of defect detection by comprehensively considering inherent characteristics such as the shape and size of the prediction area.

[0102] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of a defect detection device for a printed circuit board provided in one embodiment of the present application. The device can implement all or part of the defect detection device for a printed circuit board through software, hardware, or a combination of both. The device 8 includes:

[0103] A data acquisition module 81 is used to obtain a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module;

[0104] A feature extraction module 82 is used to input the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales;

[0105] A feature sampling module 83 is used to input the backbone feature maps of several scales into the feature fusion upsampling module in the neck network to perform feature fusion upsampling to obtain a feature fusion upsampling map;

[0106] A feature fusion module 84 is used to input the feature fusion up-sampling graphs into the convolution connection feature fusion module to perform convolution connection feature fusion, so as to obtain convolution connection feature fusion graphs of several scales;

[0107] The defect detection module 85 is used to input the convolution connection feature fusion graphs of several scales into the detection network for target detection, so as to obtain the defect detection result of the printed circuit board image to be detected.

[0108] In an embodiment of the present application, a printed circuit board image to be detected and a preset defect detection model are obtained through a data acquisition module, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolutional connection feature fusion module; through a feature extraction module, the printed circuit board image to be detected is input into the backbone network for feature extraction to obtain backbone feature maps of several scales; through a feature sampling module, the backbone feature maps of several scales are input into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map; through a feature fusion module, the feature fusion upsampling map is respectively input into the convolutional connection feature fusion module for convolutional connection feature fusion to obtain convolutional connection feature fusion maps of several scales; through a defect detection module, the convolutional connection feature fusion maps of several scales are input into the detection network for target detection to obtain a defect detection result of the printed circuit board image to be detected. The multi-scale backbone feature map of the constructed printed circuit board image is subjected to feature fusion upsampling and convolution connection feature fusion, which reduces the computational burden and improves the accuracy and efficiency of feature extraction of small targets for printed circuit board defect detection, thereby achieving comprehensive and sufficient defect detection of printed circuit board images and improving the accuracy and efficiency of printed circuit board defect detection.

[0109] Please refer to Fig. 9 , Fig. 9 The computer device 9 is a schematic diagram of a structure of a computer device provided in an embodiment of the present application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device may store a plurality of instructions, which are suitable for being loaded and executed by the processor 91 Figures 1 to 7 The method steps shown in the figure can be found in the specific execution process. Figures 1 to 7 The specific description shown will not be repeated here.

[0110] Among them, the processor 91 may include one or more processing cores. The processor 91 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the defect detection device 8 of the printed circuit board by running or executing instructions, programs, code sets or instruction sets stored in the memory 92, and calling the data in the memory 92. Optionally, the processor 91 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 91 can integrate one or more combinations of a central processing unit 91 (Central Processing Unit, CPU), a graphics processing unit 91 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 91, but implemented by a single chip.

[0111] Among them, the memory 92 may include a random access memory 92 (Random Access Memory, RAM), and may also include a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also be optionally at least one storage device located away from the aforementioned processor 91.

[0112] The present application also provides a storage medium that can store multiple instructions, which are suitable for the processor to load and execute the above-mentioned Figures 1 to 7 The method steps shown in the figure can be found in the specific execution process. Figures 1 to 7 The specific description shown will not be repeated here.

[0113] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0116] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0120] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A defect detection method for a printed circuit board, characterized in that: The following steps are involved: Obtaining a printed circuit board image to be inspected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module; Inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales; Inputting backbone feature maps of several scales into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain feature fusion upsampling maps; The feature fusion up-sampling graphs are respectively input into the convolutional connection feature fusion module for convolutional connection feature fusion to obtain convolutional connection feature fusion graphs of several scales; The convolutional connection feature fusion graphs of several scales are input into the detection network for target detection to obtain defect detection results of the printed circuit board image to be detected.

2. The defect detection method for a printed circuit board according to claim 1, characterized in that: The backbone network includes a downsampling module, a convolution fusion module and a spatial pyramid pooling module connected in sequence; the convolution fusion module includes a plurality of convolution fusion submodules connected in sequence, and the convolution fusion submodule includes a convolution unit and a depth-separable convolution unit connected in sequence; The step of inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales comprises the following steps: The printed circuit board image to be detected is sequentially down-sampled by the down-sampling module to obtain a feature extraction map; The feature extraction map is used as the first input feature map of the first convolution fusion submodule of the convolution fusion module, and is processed by the convolution unit and the depth-separable convolution unit in sequence to obtain the first output feature map of the first convolution fusion submodule, and the first output feature map is used as the first input feature map of the next convolution fusion submodule, and the process is repeated until the first output feature map of the last convolution fusion submodule is obtained; The first output feature map of the last convolution fusion submodule is pooled through the spatial pyramid pooling module to obtain a pooled processing map as the backbone feature map of the last scale, and the first output feature maps of other convolution fusion submodules are used as the backbone feature maps of the corresponding scales to obtain backbone feature maps of several scales.

3. The defect detection method for a printed circuit board according to claim 2, characterized in that: The feature fusion upsampling module includes a plurality of sequentially connected feature fusion upsampling submodules, wherein the feature fusion upsampling submodule includes a sequentially connected convolution unit, a DySample upsampling unit, a connection unit, a feature modeling unit, and a CBAM attention mechanism unit; The step of inputting backbone feature maps of several scales into a feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map comprises the following steps: The backbone feature map of the last scale is used as the second input feature map of the first feature fusion upsampling submodule, and the second input feature map is processed by the convolution unit and the DySample upsampling unit in sequence to obtain the second input feature map after upsampling; Perform feature connection on the second input feature map after upsampling and the backbone feature map of the previous scale of the second input feature map through a connection unit to obtain a first feature connection map; The first feature connection graph is subjected to attention extraction by the feature modeling unit and the CBAM attention mechanism unit to obtain a first attention feature graph as the second output feature graph of the first feature fusion upsampling submodule; The second output feature map of the first feature fusion upsampling submodule is used as the second input feature map of the next feature fusion upsampling submodule, and the feature fusion upsampling is repeated until the second output feature map of the last feature fusion upsampling submodule is obtained. The second output feature map of the last feature fusion upsampling submodule is used as the feature fusion upsampling map to obtain the feature fusion upsampling map.

4. The defect detection method for a printed circuit board according to claim 3, characterized in that: The convolution connection feature fusion module includes a plurality of sequentially connected convolution connection feature fusion submodules, wherein the convolution connection feature fusion submodule includes a sequentially connected convolution unit, a connection unit, a linear time series modeling unit and a CBAM attention mechanism unit; The feature fusion up-sampling graphs are respectively input into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion graphs of several scales, including the steps of: The feature fusion up-sampled map is used as the third input feature map of the first convolution connection feature fusion submodule, and the third input feature map is convolved by the convolution unit to obtain the third input feature map after the convolution process; the backbone feature map of a scale on the third input feature map is used as the to-be-connected map, and the third input feature map after the convolution process and the to-be-connected map are feature-connected by the connection unit to obtain the second feature connection map; The second feature connection graph is subjected to feature modeling by the linear time series modeling unit to obtain a linear time series modeling graph; the linear time series modeling graph is subjected to attention extraction by the CBAM attention mechanism unit to obtain a second attention feature graph as the third output feature graph of the first convolution connection feature fusion submodule; The third output feature map of the first convolutional connection feature fusion submodule is used as the third input feature map of the next convolutional connection feature fusion submodule and the map to be connected, and the convolutional connection feature fusion is repeated until the linear time series modeling map output by the linear time series modeling unit of the last convolutional connection feature fusion submodule is obtained. The linear time series modeling map is used as the convolutional connection feature fusion map to obtain convolutional connection feature fusion maps of several scales.

5. The defect detection method for a printed circuit board according to claim 4, characterized in that: The linear time series modeling unit includes a linear projection unit, a convolution unit and a state space unit; the second feature connection graph is subjected to feature modeling by the linear time series modeling unit to obtain a linear time series modeling graph, including the steps of: Inputting the second feature connection map into the linear projection unit for feature mapping to obtain a feature map, inputting the feature map into the convolution unit for convolution to obtain an intermediate convolution feature map; performing nonlinear processing on the intermediate convolution feature map to obtain the intermediate convolution feature map after nonlinear processing; The intermediate convolution feature map after the nonlinear processing is input as the fourth input feature map of the state space unit, and a fourth output feature map of the state space unit is obtained as the state space feature map according to a preset state space calculation algorithm, wherein the state space calculation algorithm is: h k =Ah k-1 +Bx k the k =Ch k In the formula, h k is the hidden layer state diagram for the next time step, h k-1 is the hidden layer state diagram of the current time step, x k is the fourth input feature map for the next time step, y k is the fourth output feature map of the next time step, A, B, and C are the state transfer matrix, input matrix, and output matrix, respectively, and the state transfer matrix, input matrix, and output matrix are all matrices updated by back propagation during the training process of the linear time series modeling unit; The state-space feature graph and the feature map are subjected to nonlinear processing to obtain the state-space feature graph and the feature map after nonlinear processing; the state-space feature graph and the feature map after nonlinear processing are subjected to residual connection to obtain a residual connection feature map; the residual connection feature map is input into the linear projection unit for feature mapping to obtain a linear time series modeling graph.

6. The defect detection method for a printed circuit board according to claim 5, characterized in that: The convolution connection feature fusion graphs of several scales are input into the detection network for target detection to obtain defect detection results of the printed circuit board image to be detected, including the steps of: According to the convolutional connection feature fusion map and the preset detector, several predicted areas of the printed circuit board image to be detected and the coordinate information and defect category information of the several predicted areas are obtained as the defect detection result of the printed circuit board image to be detected; wherein the coordinate information includes the center point coordinate parameters, width parameters and height parameters of the predicted area, which are used to indicate the position and size of the predicted area, and the defect category information is used to indicate the defect category of the predicted area.

7. The defect detection method for a printed circuit board according to claim 6, characterized in that: It also includes the steps of: training the defect detection model; The training of the defect detection model comprises the steps of: Obtaining a plurality of sample printed circuit board images and a plurality of defect label results of the sample printed circuit board images, wherein the defect label results include coordinate information of a plurality of label regions and defect category information; Inputting a number of the sample printed circuit board images into the defect detection model to be trained, obtaining defect detection results of the sample printed circuit board images; obtaining a regression loss value according to the defect detection results of the sample printed circuit board images, defect label results and a preset Shape-IoU loss function, and training the defect detection model to be trained according to the regression loss value, wherein the Shape-IoU loss function is: L Shape-IoU =1-IoU+distance shape +0.5×Ω shape Where, distance shape is the point distance, scale is the scale factor, h gt is the height parameter of the label area, w gt is the width parameter of the label area, x c is the horizontal coordinate parameter in the center point coordinate parameter of the prediction area, is the horizontal coordinate parameter in the center coordinate parameter of the label area, and c represents (x c ,y c )as well as The hypotenuse length of the minimum area between is the ordinate parameter in the coordinate parameter of the center point of the label area, Ω shape is the shape similarity loss value, wt is the weight of the shape similarity loss, Θ is the shape similarity loss index, L Shape-IoU is the regression loss value, IoU is the intersection over union ratio, which is the ratio of the intersection and union of the prediction area and the label area.

8. A defect detection device for a printed circuit board, characterized in that: include: A data acquisition module, used to obtain a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, and the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module; A feature extraction module, used for inputting the printed circuit board image to be detected into the backbone network for feature extraction, and obtaining backbone feature maps of several scales; A feature sampling module, used for inputting backbone feature maps of several scales into a feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map; A feature fusion module, used for inputting the feature fusion up-sampling graphs into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion graphs of several scales; The defect detection module is used to input the convolution connection feature fusion graphs of several scales into the detection network for target detection to obtain the defect detection result of the printed circuit board image to be detected.

9. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the printed circuit board defect detection method as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the printed circuit board defect detection method according to any one of claims 1 to 7 are implemented.

Images