Defect detection method, device and equipment of printed circuit board and storage medium

Abstract

The present application relates to the field of defect detection, and particularly relates to a printed circuit board defect detection method, device, equipment and storage medium, feature fusion upsampling and convolution connection feature fusion are performed on the multi-scale backbone feature map of the constructed printed circuit board image, the calculation burden is reduced, the accuracy and efficiency of small target feature extraction are improved, so as to perform printed circuit board defect detection, comprehensive and sufficient defect detection of the printed circuit board image is realized, and the accuracy and efficiency of printed circuit board defect detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of defect detection, and particularly relates to a printed circuit board defect detection method and device, a device and a storage medium. BACKGROUND

[0002] Due to process problems, there are manufacturing defect problems in the manufacturing of printed circuit boards (PCB). Common PCB defects mainly include hole defects, circuit defects, pad defects, solder mask layer defects and the like. These defects can cause unstable connection of the circuit board, interference of inter-board signal transmission, and thus cause problems such as shortened service life of assembled electronic products, functional failure, and unstable operation of electronic products. Therefore, strict quality control is needed at each production stage through various detection methods.

[0003] At present, traditional printed circuit board defect detection mainly includes manual detection (visual inspection), automatic optical detection, X-ray detection and the like. However, the traditional detection has the characteristics of low efficiency, long time consumption and high missed detection rate. With the continuous progress of computer vision technology and the continuous upgrading of hardware resources (such as GPU), PCB defect detection algorithms based on computer vision technology are widely used in PCB defect detection. Although the detection accuracy problem can be solved, the detection speed is slow and cannot meet the rapid detection needs of a large number of PCBs passing through the production line in the factory production scene. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a printed circuit board defect detection method, device, equipment and storage medium, which can reduce the computational burden, improve the accuracy and efficiency of small target feature extraction, and realize comprehensive and sufficient defect detection of printed circuit board images, thereby improving the accuracy and efficiency of printed circuit board defect detection.

[0005] In a first aspect, the present application provides a printed circuit board defect detection method, including the following steps:

[0006] obtaining a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model includes a backbone network, a neck network and a detection network, the neck network includes a feature fusion upsampling module and a convolution connection feature fusion module;

[0007] inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales;

[0008] The backbone feature maps of several scales are input into a feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map;

[0009] The feature fusion upsampling map is input into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion maps of several scales.

[0010] The convolution connection feature fusion maps of several scales are input into the detection network for target detection to obtain a defect detection result of the printed circuit board image to be detected.

[0011] In a second aspect, an embodiment of the present application provides a printed circuit board defect detection device, comprising:

[0012] A data acquisition module is configured to obtain a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model comprises a backbone network, a neck network, and a detection network, the neck network comprises a feature fusion upsampling module and a convolution connection feature fusion module.

[0013] A feature extraction module is configured to input the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales.

[0014] A feature sampling module is configured to input the backbone feature maps of several scales into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain a feature fusion upsampling map.

[0015] A feature fusion module is configured to input the feature fusion upsampling map into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion maps of several scales.

[0016] A defect detection module is configured to input the convolution connection feature fusion maps of several scales into the detection network for target detection to obtain a defect detection result of the printed circuit board image to be detected.

[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the printed circuit board defect detection method according to the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores a computer program, when the computer program is executed by a processor, the steps of the printed circuit board defect detection method according to the first aspect are implemented.

[0019] In the embodiment of the present application, a defect detection method, device and equipment of a printed circuit board and a storage medium are provided. The multi-scale backbone feature maps of the constructed printed circuit board image are subjected to feature fusion upsampling and convolution connection feature fusion, thereby reducing the calculation burden, improving the accuracy and efficiency of feature extraction of small targets, and performing defect detection of the printed circuit board, achieving comprehensive and sufficient defect detection of the printed circuit board image, and improving the accuracy and efficiency of the printed circuit board defect detection.

[0020] In order to better understand and implement, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of a defect detection method of a printed circuit board provided by an embodiment of the present application is shown in the figure.

[0022] Figure 2 A flowchart of S2 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in the figure.

[0023] Figure 3 A flowchart of S3 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in the figure.

[0024] Figure 4 A flowchart of S4 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in the figure.

[0025] Figure 5 A flowchart of S42 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in the figure.

[0026] Figure 6 A flowchart of S5 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in the figure.

[0027] Figure 7 A flowchart of S6 in the defect detection method of the printed circuit board provided by another embodiment of the present application is shown in the figure.

[0028] Figure 8 A structural diagram of a defect detection device of a printed circuit board provided by an embodiment of the present application is shown in the figure.

[0029] Figure 9 A structural diagram of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to any embodiment of the application, unless specifically stated otherwise. It is to be understood that other embodiments can be utilized, and structural or procedural changes can be made without departing from the scope of the present application. Therefore, the following detailed description is not intended to be limiting.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] Please refer to Figure 1 , Figure 1 The flowchart of the defect detection method of the printed circuit board provided by an embodiment of the present application, the method comprises the following steps:

[0034] S1: obtaining the printed circuit board image to be detected and the preset defect detection model.

[0035] The execution subject of the defect detection method of the printed circuit board is a defect detection device of the printed circuit board (hereinafter referred to as a detection device). In an optional embodiment, the detection device can be a computer device, which can be a server, or a server cluster formed by multiple computer devices.

[0036] In this embodiment, the detection device can obtain the printed circuit board image to be detected input by the user, or obtain the printed circuit board image to be detected from the preset database.

[0037] The detection equipment obtains a preset defect detection model, wherein the photovoltaic panel defect detection model is an improved Yolov5 model, which is a target detection model that can solve various problems in the actual operation of printed circuit board defect detection and balance the calculation amount and accuracy, and includes a backbone network, a neck network, and a detection network (YoloHead). The neck network includes a feature fusion upsampling module and a convolution connection feature fusion module.

[0038] S2: input the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales.

[0039] In this embodiment, the detection equipment inputs the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps of several scales.

[0040] The backbone network includes a downsampling module, a convolution fusion module, and a spatial pyramid pooling module connected in sequence; the convolution fusion module includes a plurality of convolution fusion sub-modules connected in sequence, and the convolution fusion sub-module includes a convolution unit and a depth separable convolution unit connected in sequence; please refer to Figure 2 , Figure 2 The flowchart of S2 in the printed circuit board defect detection method provided by an embodiment of the present application includes steps S21-S23, and the details are as follows:

[0041] S21: sequentially perform downsampling processing on the printed circuit board image to be detected through the downsampling module to obtain a feature extraction map.

[0042] The downsampling module adopts a focus module, which is a convolutional neural network layer for feature extraction, and is used to compress and combine the information in the input feature map to extract higher-level feature representations.

[0043] In this embodiment, the detection equipment sequentially performs downsampling processing on the printed circuit board image to be detected through the downsampling module to obtain a feature extraction map, so as to reduce the calculation amount and parameter amount.

[0044] S22: take the feature extraction map as the first input feature map of the first convolution fusion sub-module of the convolution fusion module, sequentially perform processing through the convolution unit and the depth separable convolution unit to obtain the first output feature map of the first convolution fusion sub-module, take the first output feature map as the first input feature map of the next convolution fusion sub-module, and repeat the processing until the first output feature map of the last convolution fusion sub-module is obtained.

[0045] The convolution unit is a convolution layer, and the depth separable convolution unit is a unit composed of three C3 (csp bottleneck with 3 conv) blocks.

[0046] In this embodiment, the detection device takes the feature extraction graph as the first input feature graph of the first convolution fusion submodule of the convolution fusion module, sequentially processes the convolution unit and the depth separable convolution unit to obtain the first output feature graph of the first convolution fusion submodule, takes the first output feature graph as the first input feature graph of the next convolution fusion submodule, and repeatedly processes until the first output feature graph of the last convolution fusion submodule is obtained.

[0047] S23: performing pooling processing on the first output feature graph of the last convolution fusion submodule through the spatial pyramid pooling module to obtain a pooling processing graph as a backbone feature graph of a last scale, and taking the first output feature graphs of the other convolution fusion submodules as backbone feature graphs of corresponding scales to obtain backbone feature graphs of several scales.

[0048] In this embodiment, the detection device performs pooling processing on the first output feature graph of the last convolution fusion submodule through the spatial pyramid pooling module to obtain a pooling processing graph as a backbone feature graph of a last scale, and takes the first output feature graphs of the other convolution fusion submodules as backbone feature graphs of corresponding scales to obtain backbone feature graphs of several scales.

[0049] Specifically, the detection device performs convolution on the first output feature graph of the last convolution fusion submodule to extract feature information in the horizontal direction and the vertical direction, generates an attention feature graph through convolution transformation, and finally performs multiplication operation on the attention feature graph and the first output feature graph of the last convolution fusion submodule to obtain a pooling processing graph as a backbone feature graph of a last scale, so as to sufficiently extract detailed information of the printed circuit board image to be detected and improve the accuracy of defect detection.

[0050] S3: inputting the backbone feature graphs of the several scales into the feature fusion upsampling module in the neck network to perform feature fusion upsampling to obtain a feature fusion upsampling graph.

[0051] In this embodiment, the detection device inputs the backbone feature graphs of the several scales into the feature fusion upsampling module in the neck network to perform feature fusion upsampling to obtain a feature fusion upsampling graph.

[0052] The feature fusion upsampling module comprises a plurality of feature fusion upsampling sub-modules connected in sequence, and each feature fusion upsampling sub-module comprises a convolution unit, a DySample upsampling unit, a connection unit, a feature modeling unit and a CBAM (Convolutional Block Attention Module) attention mechanism unit connected in sequence. Figure 3 , Figure 3 The flowchart of S3 in the defect detection method of the printed circuit board provided in an embodiment of the present application comprises steps S31-S33, and details are as follows.

[0053] S31: The backbone feature map of the last scale is taken as the second input feature map of the first feature fusion upsampling sub-module, and the second input feature map is sequentially processed by the convolution unit and the DySample upsampling unit to obtain the second input feature map after upsampling processing.

[0054] The DySample module optimizes the generation of the offset in upsampling, creates a sampling set by a sampling generator and is used for dynamic upsampling, so as to pay more attention to the defect information in the image and reduce the noise influence.

[0055] In the embodiment, the detection device takes the backbone feature map of the last scale as the second input feature map of the first feature fusion upsampling sub-module, and sequentially processes the second input feature map by the convolution unit and the DySample upsampling unit to obtain the second input feature map after upsampling processing.

[0056] S32: The second input feature map after upsampling processing and the backbone feature map of the last scale of the second input feature map are connected by the connection unit to obtain a first feature connection map; the first feature connection map is subjected to attention extraction by the feature modeling unit and the CBAM attention mechanism unit to obtain a first attention feature map as the second output feature map of the first feature fusion upsampling sub-module.

[0057] In the embodiment, the detection device connects the second input feature map after upsampling processing and the backbone feature map of the last scale of the second input feature map by the connection unit to obtain a first feature connection map.

[0058] The feature modeling unit is a unit composed of three C3 (csp bottleneck with 3 conv) blocks, and the feature modeling unit of the last feature fusion upsampling module adopts a linear time series modeling unit (Mamba). The main advantage of the linear time series modeling unit is its excellent performance on long sequence tasks and low computational complexity. Through discretization and convolution, the calculation efficiency is improved, and the model's ability to capture long-distance information is improved.

[0059] The CBAM attention mechanism unit is a lightweight and effective attention mechanism. By introducing channel attention and spatial attention, the model can more accurately capture and utilize key information.

[0060] The detection device extracts attention from the first feature connection graph through the feature modeling unit and the CBAM attention mechanism unit to obtain a first attention feature graph as the second output feature graph of the first feature fusion upsampling sub-module. The attention mechanism is introduced to extract the feature information in the printed circuit board image after the feature modeling.

[0061] S33: The second output feature graph of the first feature fusion upsampling sub-module is taken as the second input feature graph of the next feature fusion upsampling sub-module, and the feature fusion upsampling is repeated until the second output feature graph of the last feature fusion upsampling sub-module is obtained. The second output feature graph of the last feature fusion upsampling sub-module is taken as the feature fusion upsampling graph, and the feature fusion upsampling graph is obtained.

[0062] In this embodiment, the detection device takes the second output feature graph of the first feature fusion upsampling sub-module as the second input feature graph of the next feature fusion upsampling sub-module, and repeats the feature fusion upsampling until the second output feature graph of the last feature fusion upsampling sub-module is obtained. The second output feature graph of the last feature fusion upsampling sub-module is taken as the feature fusion upsampling graph, and the feature fusion upsampling graph is obtained.

[0063] S4: The feature fusion upsampling graph is input into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion graphs of several scales.

[0064] In this embodiment, the detection device inputs the feature fusion upsampling graph into the convolution connection feature fusion module for convolution connection feature fusion to obtain convolution connection feature fusion graphs of several scales.

[0065] The convolution connection feature fusion module comprises a plurality of convolution connection feature fusion sub-modules connected in sequence, and the convolution connection feature fusion sub-module comprises convolution units, connection units, linear time series modeling units and CBAM attention mechanism units connected in sequence. Figure 4 , Figure 4 The flowchart of S4 in the defect detection method of the printed circuit board provided in an embodiment of the present application comprises steps S41-S43, and details are as follows.

[0066] S41: The feature fusion up-sampling map is taken as the third input feature map of the first convolution connection feature fusion sub-module, the third input feature map is subjected to convolution processing through the convolution unit to obtain the third input feature map after convolution processing, and a backbone feature map of the next scale of the third input feature map is taken as a to-be-connected map, and the third input feature map after convolution processing and the to-be-connected map are subjected to feature connection through the connection unit to obtain a second feature connection map.

[0067] In the embodiment, the detection device takes the feature fusion up-sampling map as the third input feature map of the first convolution connection feature fusion sub-module, subjects the third input feature map to convolution processing through the convolution unit to obtain the third input feature map after convolution processing.

[0068] The detection device takes a backbone feature map of the next scale of the third input feature map as a to-be-connected map, and subjects the third input feature map after convolution processing and the to-be-connected map to feature connection through the connection unit to obtain a second feature connection map.

[0069] S42: The second feature connection map is subjected to feature modeling through the linear time series modeling unit to obtain a linear time series modeling map, and the linear time series modeling map is subjected to attention extraction through the CBAM attention mechanism unit to obtain a second attention feature map as the third output feature map of the first convolution connection feature fusion sub-module.

[0070] In the embodiment, the detection device subjects the second feature connection map to feature modeling through the linear time series modeling unit to obtain a linear time series modeling map. A linear time series modeling method of selecting a state space is adopted to model the second feature connection map, to construct a long-distance dependency relationship and maintain the linear complexity of the feature map, thereby reducing the calculation burden and improving the accuracy and efficiency of feature extraction of small targets.

[0071] The detection equipment performs attention extraction on the linear time sequence modeling graph through the CBAM attention mechanism unit to obtain a second attention feature map as a third output feature map of a first convolution connection feature fusion sub-module, introduces an attention mechanism, performs attention extraction on the data after feature modeling, fully extracts feature information in the printed circuit board image, and improves the accuracy and efficiency of printed circuit board defect detection.

[0072] The linear time sequence modeling unit includes a linear projection unit, a convolution unit, and a state space unit; please refer to Figure 5 , Figure 5 The flowchart of S42 in the printed circuit board defect detection method provided by an embodiment of the application includes steps S421-S423, and specifically as follows:

[0073] S421: input the second feature connection graph into the linear projection unit for feature mapping to obtain a feature mapping graph, input the feature mapping graph into the convolution unit for convolution to obtain an intermediate convolution feature map, and perform nonlinear processing on the intermediate convolution feature map to obtain the intermediate convolution feature map after nonlinear processing.

[0074] In order to project the feature tensor to a higher dimension for processing to capture more detailed features, in the embodiment, the detection equipment inputs the second feature connection graph into the linear projection unit for feature mapping to obtain a feature mapping graph.

[0075] The detection equipment inputs the feature mapping graph into the convolution unit for convolution to obtain an intermediate convolution feature map, and performs nonlinear processing on the intermediate convolution feature map to obtain the intermediate convolution feature map after nonlinear processing, so as to capture the information of adjacent tokens before inputting the feature mapping graph into the state space unit, and improve the accuracy of feature modeling.

[0076] S422: input the intermediate convolution feature map after nonlinear processing as a fourth input feature map of the state space unit, obtain a fourth output feature map of the state space unit as the state space feature map according to a preset state space calculation algorithm.

[0077] The state space calculation algorithm is:

[0078] h k =Ah k-1 +Bx k

[0079] y k =Ch k

[0080] In the formula, h kis a hidden layer state map of a next time step, h k-1 is a hidden layer state map of a current time step, x k is a fourth input feature map of a next time step, y k is a fourth output feature map of a next time step, A, B, C are respectively a state transition matrix, an input matrix and an output matrix, and the state transition matrix, the input matrix and the output matrix are matrices updated by back propagation during training of the linear time series modeling unit.

[0081] In this embodiment, the detection device inputs the intermediate convolution feature map after the nonlinear processing as the fourth input feature map of the state space unit, obtains the fourth output feature map of the state space unit according to a preset state space calculation algorithm, and takes the fourth output feature map of the state space unit as the state space feature map.

[0082] S423: The state space feature map and the feature mapping map are nonlinearly processed to obtain a nonlinearly processed state space feature map and a nonlinearly processed feature mapping map. The nonlinearly processed state space feature map and the nonlinearly processed feature mapping map are residual connected to obtain a residual connection feature map. The residual connection feature map is input into the linear projection unit for feature mapping to obtain a linear time series modeling map.

[0083] In this embodiment, the detection device nonlinearly processes the state space feature map and the feature mapping map to obtain a nonlinearly processed state space feature map and a nonlinearly processed feature mapping map. The nonlinearly processed state space feature map and the nonlinearly processed feature mapping map are residual connected to obtain a residual connection feature map. The residual connection feature map is input into the linear projection unit for feature mapping to obtain a linear time series modeling map.

[0084] S43: The third output feature map of the first convolution connection feature fusion sub-module is taken as the third input feature map of the next convolution connection feature fusion sub-module and the to-be-connected map. Convolution connection feature fusion is repeatedly performed until a linear time series modeling map output by the linear time series modeling unit of the last convolution connection feature fusion sub-module is obtained. The linear time series modeling map is taken as a convolution connection feature fusion map to obtain convolution connection feature fusion maps of several scales.

[0085] In this embodiment, the detection device takes the third output feature map of the first convolution connection feature fusion sub-module as the third input feature map of the next convolution connection feature fusion sub-module and the to-be-connected map. Convolution connection feature fusion is repeatedly performed until a linear time series modeling map output by the linear time series modeling unit of the last convolution connection feature fusion sub-module is obtained. The linear time series modeling map is taken as a convolution connection feature fusion map to obtain convolution connection feature fusion maps of several scales.

[0086] S5: inputting the convolution connection feature fusion map of the plurality of scales into the detection network for target detection to obtain a defect detection result of the printed circuit board image to be detected.

[0087] In this embodiment, the detection device inputs the convolution connection feature fusion map of the plurality of scales into the detection network for target detection to obtain a defect detection result of the printed circuit board image to be detected.

[0088] Referring to Figure 6 , Figure 6 A flowchart of S5 in the defect detection method of the printed circuit board provided by an embodiment of the present application is shown in FIG. 5, which includes the following step S51:

[0089] S51: obtaining, according to the convolution connection feature fusion map and a preset detector, a plurality of prediction regions of the printed circuit board image to be detected and coordinate information and defect category information of the plurality of prediction regions as the defect detection result of the printed circuit board image to be detected.

[0090] In this embodiment, the detection device obtains, according to the convolution connection feature fusion map and a preset detector, a plurality of prediction regions of the printed circuit board image to be detected and coordinate information and defect category information of the plurality of prediction regions as the defect detection result of the printed circuit board image to be detected, wherein the coordinate information includes a center point coordinate parameter, a width parameter and a height parameter of the prediction region, which are used to indicate the position and size of the prediction region, and the defect category information is used to indicate the defect category of the prediction region.

[0091] The linear time series modeling method of selecting state space is adopted to perform feature fusion upsampling and convolution connection feature fusion on the constructed multi-scale backbone feature map of the printed circuit board image, thereby reducing the calculation burden, improving the accuracy and efficiency of feature extraction of small targets, and performing defect detection on the printed circuit board, achieving comprehensive and sufficient defect detection on the printed circuit board image, and improving the accuracy and efficiency of the defect detection of the printed circuit board.

[0092] In an optional embodiment, the method further includes the following step S7: training the defect detection model. Referring to Figure 7 , Figure 7 A flowchart of S6 in the defect detection method of the printed circuit board provided by another embodiment of the present application is shown in FIG. 6, which includes the following steps S61-S62:

[0093] S61: obtaining a plurality of sample printed circuit board images and defect label results of the plurality of sample printed circuit board images.

[0094] In the embodiment, the detection device obtains a plurality of sample printed circuit board images and a plurality of defect label results of the sample printed circuit board images, wherein the defect label results include coordinate information of a plurality of label regions and defect category information.

[0095] Specifically, the detection device uses the public dataset PKU-Market-PCB of the Artificial Intelligence Laboratory of Peking University, and the detection device performs data enhancement on the pictures in the PKU-Market-PCB dataset through expansion, brightness transformation, salt and pepper noise, Gaussian noise, translation and rotation to obtain a plurality of sample printed circuit board images.

[0096] S62: inputting the plurality of sample printed circuit board images into a defect detection model to be trained to obtain a plurality of defect detection results of the sample printed circuit board images; obtaining a regression loss value according to the defect detection results of the plurality of sample printed circuit board images, the defect label results and a preset Shape-IoU loss function, and training the defect detection model to be trained according to the regression loss value.

[0097] The Shape-IoU loss function is:

[0098]

[0099] L Shape-IoU = 1 - IoU + distance shape + 0.5 x Ω shape

[0100] In the formula, distance shape is a point distance, scale is a scale factor, h gt is a height parameter of a label region, w gt is a width parameter of a label region, x c is a horizontal coordinate parameter in a center point coordinate parameter of a prediction region, is a horizontal coordinate parameter in a center point coordinate parameter of a label region, c represents the minimum region diagonal length between (x c , y c ) and , and is a vertical coordinate parameter in a center point coordinate parameter of a label region, Ω shape is a shape similarity loss value, wt is a weight of the shape similarity loss, Θ is a shape similarity loss index, L Shape-IoU is a regression loss value, IoU is an intersection over union, and the ratio of the intersection and the union of the prediction region and the label region.

[0101] In the embodiment, the detection device inputs a plurality of sample printed circuit board images into a defect detection model to be trained to obtain defect detection results of the plurality of sample printed circuit board images; according to the defect detection results of the plurality of sample printed circuit board images, defect label results and a preset Shape-IoU loss function, a regression loss value is obtained, the defect detection model to be trained is trained according to the regression loss value, and the accuracy and efficiency of defect detection are improved by comprehensively considering inherent features such as shapes and sizes of predicted regions.

[0102] Please refer to Figure 8 , Figure 8 The structure diagram of the defect detection device of a printed circuit board provided by an embodiment of the present application, which can realize all or part of the defect detection device of the printed circuit board through software, hardware or a combination of both. The device 8 comprises:

[0103] A data acquisition module 81 is configured to obtain a printed circuit board image to be detected and a preset defect detection model, wherein the defect detection model comprises a backbone network, a neck network and a detection network, and the neck network comprises a feature fusion upsampling module and a convolution connection feature fusion module.

[0104] A feature extraction module 82 is configured to input the printed circuit board image to be detected into the backbone network to perform feature extraction and obtain backbone feature maps of a plurality of scales.

[0105] A feature sampling module 83 is configured to input the backbone feature maps of a plurality of scales into the feature fusion upsampling module in the neck network to perform feature fusion upsampling and obtain a feature fusion upsampling map.

[0106] A feature fusion module 84 is configured to input the feature fusion upsampling map into the convolution connection feature fusion module respectively to perform convolution connection feature fusion and obtain convolution connection feature fusion maps of a plurality of scales.

[0107] A defect detection module 85 is configured to input the convolution connection feature fusion maps of a plurality of scales into the detection network to perform target detection and obtain defect detection results of the printed circuit board image to be detected.

[0108] In the embodiment of the present application, the printed circuit board image to be detected and a preset defect detection model are obtained through the data acquisition module, wherein the defect detection model comprises a backbone network, a neck network and a detection network, the neck network comprises a feature fusion upsampling module and a convolution connection feature fusion module; the printed circuit board image to be detected is input into the backbone network for feature extraction through the feature extraction module, and a plurality of scale backbone feature maps are obtained; the plurality of scale backbone feature maps are input into the feature fusion upsampling module in the neck network for feature fusion upsampling through the feature sampling module, and a feature fusion upsampling map is obtained; the feature fusion upsampling map is input into the convolution connection feature fusion module for convolution connection feature fusion through the feature fusion module, and a plurality of scale convolution connection feature fusion maps are obtained; the plurality of scale convolution connection feature fusion maps are input into the detection network for target detection through the defect detection module, and a defect detection result of the printed circuit board image to be detected is obtained. The multi-scale backbone feature maps of the printed circuit board image are subjected to feature fusion upsampling and convolution connection feature fusion, thereby reducing the calculation burden, improving the accuracy and efficiency of feature extraction of small targets, and being used for defect detection of the printed circuit board, realizing comprehensive and sufficient defect detection of the printed circuit board image, and improving the accuracy and efficiency of the printed circuit board defect detection.

[0109] Please refer to Figure 9 , Figure 9 The structural schematic diagram of the computer device provided in an embodiment of the present application, the computer device 9 comprises a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device can store a plurality of instructions, the instructions are suitable for being loaded and executed by the processor 91 to execute the method steps shown in the above Figures 1 to 7 , the specific execution process can be referred to the specific description shown in Figures 1 to 7 , which will not be described here.

[0110] The processor 91 may include one or more processing cores. The processor 91 connects to various parts within the server using various interfaces and lines. It executes various functions and processes data of the printed circuit board defect detection device 8 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 92, and by calling data from the memory 92. Optionally, the processor 91 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 91 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 91.

[0111] The memory 92 may include random access memory (RAM) or read-only memory. Optionally, the memory 92 may include a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 92 may also be at least one storage device located remotely from the aforementioned processor 91.

[0112] This application embodiment also provides a storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1 to 7 The method steps shown can be found in the following document for detailed execution process. Figures 1 to 7 The specific details shown will not be repeated here.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific name of each functional unit and module is only for the convenience of mutual distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0114] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0115] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the algorithm. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of the present application.

[0116] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal device embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0117] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0118] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0119] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program, when executed by a processor, can implement the steps of each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form.

[0120] The present application is not limited to the above-described embodiments, and various modifications or changes can be made to the present application without departing from the spirit and scope of the present application. Therefore, it is intended that the present application encompass all such modifications and changes and fall within the scope of the appended claims and their equivalents.

Claims

1. A method for defect detection in printed circuit boards, characterized in that, Includes the following steps: The system obtains an image of the printed circuit board to be inspected and a preset defect detection model. The defect detection model includes a backbone network, a neck network, and a detection network. The neck network includes a feature fusion upsampling module and a convolutional connection feature fusion module. The convolutional connection feature fusion module includes several sequentially connected convolutional connection feature fusion sub-modules, each of which includes a convolutional unit, a connection unit, a linear time series modeling unit, and a CBAM attention mechanism unit connected in sequence. The printed circuit board image to be detected is input into the backbone network for feature extraction to obtain backbone feature maps at several scales. The backbone feature maps at several scales are input into the feature fusion and upsampling module in the neck network to perform feature fusion and upsampling, thereby obtaining the feature fusion and upsampling map; The feature fusion upsampled images are input into the convolutional connection feature fusion module. The feature fusion upsampled images are used as the third input feature images of the first convolutional connection feature fusion submodule. The third input feature images are processed by convolution units to obtain the convolutional processed third input feature images. The backbone feature images of the next scale above the third input feature images are used as the images to be connected. The convolutional processed third input feature images and the images to be connected are connected by connection units to obtain the second feature connection images. The second feature connection map is processed by the linear time series modeling unit to obtain a linear time series modeling map; the linear time series modeling map is processed by the CBAM attention mechanism unit to obtain a second attention feature map, which serves as the third output feature map of the first convolutional connection feature fusion submodule. The third output feature map of the first convolutional connection feature fusion submodule is used as the third input feature map and the map to be connected in the next convolutional connection feature fusion submodule. The convolutional connection feature fusion is repeated until the linear time series modeling map output by the linear time series modeling unit of the last convolutional connection feature fusion submodule is obtained. The linear time series modeling map is used as the convolutional connection feature fusion map to obtain convolutional connection feature fusion maps at several scales. The convolutional connection feature fusion map of several scales is input into the detection network for target detection to obtain the defect detection result of the printed circuit board image to be detected.

2. The defect detection method for printed circuit boards according to claim 1, characterized in that: The backbone network includes a downsampling module, a convolutional fusion module, and a spatial pyramid pooling module connected in sequence; the convolutional fusion module includes a number of convolutional fusion sub-modules connected in sequence, and the convolutional fusion sub-module includes convolutional units and depthwise separable convolutional units connected in sequence. The step of inputting the printed circuit board image to be detected into the backbone network for feature extraction to obtain backbone feature maps at several scales includes the following steps: The printed circuit board image to be detected is sequentially passed through the downsampling module for downsampling processing to obtain a feature extraction map; The extracted feature map is used as the first input feature map of the first convolutional fusion sub-module of the convolutional fusion module. It is then processed sequentially through the convolutional unit and the depthwise separable convolutional unit to obtain the first output feature map of the first convolutional fusion sub-module. The first output feature map is used as the first input feature map of the next convolutional fusion sub-module. This process is repeated until the first output feature map of the last convolutional fusion sub-module is obtained. The first output feature map of the last convolutional fusion submodule is pooled by the spatial pyramid pooling module to obtain a pooled map, which serves as the backbone feature map for the last scale. The first output feature maps of the other convolutional fusion submodules are used as backbone feature maps for the corresponding scales to obtain backbone feature maps for several scales.

3. The defect detection method for printed circuit boards according to claim 2, characterized in that: The feature fusion upsampling module includes several sequentially connected feature fusion upsampling sub-modules, each of which includes a convolutional unit, a DySample upsampling unit, a connection unit, a feature modeling unit, and a CBAM attention mechanism unit connected in sequence. The step of inputting backbone feature maps at several scales into the feature fusion and upsampling module in the neck network for feature fusion and upsampling to obtain a feature fusion and upsampling map includes the following steps: The backbone feature map at the last scale is used as the second input feature map of the first feature fusion upsampling submodule. The second input feature map is then processed by the convolution unit and the DySample upsampling unit in sequence to obtain the upsampled second input feature map. The upsampled second input feature map and the backbone feature map of the previous scale of the second input feature map are connected by a connection unit to obtain a first feature connection map. The first feature connection map is processed by the feature modeling unit and the CBAM attention mechanism unit to extract attention, and a first attention feature map is obtained, which serves as the second output feature map of the first feature fusion upsampling submodule. The second output feature map of the first feature fusion upsampling submodule is used as the second input feature map of the next feature fusion upsampling submodule. Feature fusion upsampling is repeated until the second output feature map of the last feature fusion upsampling submodule is obtained. The second output feature map of the last feature fusion upsampling submodule is used as the feature fusion upsampling map to obtain the feature fusion upsampling map.

4. The defect detection method for printed circuit boards according to claim 1, characterized in that: The linear time series modeling unit includes a linear projection unit, a convolution unit, and a state space unit; the step of passing the second feature connection map through the linear time series modeling unit to perform feature modeling to obtain a linear time series modeling map includes the following steps: The second feature connection map is input into the linear projection unit for feature mapping to obtain a feature mapping map. The feature mapping map is then input into the convolution unit for convolution processing to obtain an intermediate convolution feature map. The intermediate convolution feature map is then subjected to nonlinear processing to obtain a nonlinearly processed intermediate convolution feature map. The intermediate convolutional feature map after nonlinear processing is used as the fourth input feature map of the state space unit. According to a preset state space calculation algorithm, the fourth output feature map of the state space unit is obtained and used as the state space feature map. The state space calculation algorithm is as follows: h k =Ah k-1 +Bx k the k =Ch k In the formula, h k For the hidden layer state graph at the next time step, h k-1 For the hidden layer state graph at the current time step, x k For the fourth input feature map at the next time step, y k The fourth output feature map for the next time step is A, B, and C, which are the state transition matrix, input matrix, and output matrix, respectively. The state transition matrix, input matrix, and output matrix are all matrices updated by backpropagation during the training of the linear time series modeling unit. The state space feature map and feature map are subjected to nonlinear processing to obtain a nonlinearly processed state space feature map and feature map. The nonlinearly processed state space feature map and feature map are then subjected to residual connection to obtain a residual connection feature map. The residual connection feature map is then input into the linear projection unit for feature mapping to obtain a linear time series modeling map.

5. The defect detection method for printed circuit boards according to claim 4, characterized in that, The step of inputting the fused convolutional connection feature map of several scales into the detection network for target detection to obtain the defect detection result of the printed circuit board image to be detected includes the following steps: Based on the convolutional connection feature fusion map and the preset detector, several predicted regions of the printed circuit board image to be detected, as well as the coordinate information and defect category information of the predicted regions, are obtained as the defect detection result of the printed circuit board image to be detected. The coordinate information includes the center point coordinate parameters, width parameters, and height parameters of the predicted regions, which are used to indicate the position and size of the predicted regions. The defect category information is used to indicate the defect category of the predicted regions.

6. The defect detection method for printed circuit boards according to claim 5, characterized in that, It also includes the step of training the defect detection model; Training the defect detection model includes the following steps: Obtain several sample printed circuit board images and several defect labeling results of the sample printed circuit board images, wherein the defect labeling results include coordinate information of several labeling regions and defect category information; Several sample printed circuit board images are input into the defect detection model to be trained to obtain defect detection results for the sample printed circuit board images. Based on the defect detection results, defect label results, and a preset Shape-IoU loss function, a regression loss value is obtained. The defect detection model to be trained is then trained based on the regression loss value, wherein the Shape-IoU loss function is: L Shape-IoU =1-IoU+distance shape +0.5×Ω shape In the formula, distance shape Where h is the distance between points, scale is the scale factor, and h is the distance between points. gt w is the height parameter of the label area. gt x is the width parameter of the label area. c The x-coordinate parameter is the center point coordinate parameter of the prediction region. The x-coordinate parameter in the center point coordinate parameters of the label area, c represents (x c y c )as well as The minimum hypotenuse length of the region between them Ω is the ordinate parameter in the coordinate parameters of the center point of the label area. shape Let wt be the shape similarity loss value, Θ be the shape similarity loss exponent, and L be the shape similarity loss value. Shape-IoU is the regression loss value, and IoU is the intersection-union ratio, which is the ratio of the intersection and union of the predicted region and the label region.

7. A defect detection device for printed circuit boards, characterized in that, include: The data acquisition module is used to obtain the printed circuit board image to be inspected and the preset defect detection model. The defect detection model includes a backbone network, a neck network, and a detection network. The neck network includes a feature fusion upsampling module and a convolutional connection feature fusion module. The convolutional connection feature fusion module includes several sequentially connected convolutional connection feature fusion sub-modules. The convolutional connection feature fusion sub-module includes a convolutional unit, a connection unit, a linear time series modeling unit, and a CBAM attention mechanism unit connected in sequence. The feature extraction module is used to input the printed circuit board image to be detected into the backbone network for feature extraction, and obtain backbone feature maps at several scales. The feature sampling module is used to input the backbone feature maps of several scales into the feature fusion upsampling module in the neck network for feature fusion upsampling to obtain the feature fusion upsampling map; The feature fusion module is used to input the feature fusion upsampled map into the convolutional connection feature fusion module, use the feature fusion upsampled map as the third input feature map of the first convolutional connection feature fusion submodule, and perform convolution processing on the third input feature map through a convolution unit to obtain the convolutional processed third input feature map; use the backbone feature map of the next scale above the third input feature map as the map to be connected, and perform feature connection on the convolutional processed third input feature map and the map to be connected through a connection unit to obtain the second feature connection map; The second feature connection map is processed by the linear time series modeling unit to obtain a linear time series modeling map; the linear time series modeling map is processed by the CBAM attention mechanism unit to obtain a second attention feature map, which serves as the third output feature map of the first convolutional connection feature fusion submodule. The third output feature map of the first convolutional connection feature fusion submodule is used as the third input feature map and the map to be connected in the next convolutional connection feature fusion submodule. The convolutional connection feature fusion is repeated until the linear time series modeling map output by the linear time series modeling unit of the last convolutional connection feature fusion submodule is obtained. The linear time series modeling map is used as the convolutional connection feature fusion map to obtain convolutional connection feature fusion maps at several scales. The defect detection module is used to input the convolutional connection feature fusion map of several scales into the detection network for target detection, and obtain the defect detection result of the printed circuit board image to be detected.

8. A computer device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor; the computer program, when executed by the processor, implements the steps of the defect detection method for a printed circuit board as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a computer program that, when executed by a processor, implements the steps of the defect detection method for printed circuit boards as described in any one of claims 1 to 6.

Images