Establishment and detection method of circuit board defect detection model

Abstract

The invention relates to image recognition, in particular to an establishment and detection method of a circuit board defect detection model. The defect detection model comprises a fusion diversified feature extraction module for extracting multi-level features from different levels and a layered attention module for gradually fusing multi-level feature information extracted from different levels; the fusion diversified feature extraction module comprises a convolution layer used for preliminary feature extraction and a plurality of feature aggregation modules used for extracting multi-level features from different levels; according to the invention, detection is carried out in the dense area and the integrated area in a multi-level feature and feature fusion mode, the image recognition precision is improved, and the recognition accuracy is guaranteed.

Description

Technical Field

[0001] The present invention relates to image recognition, and particularly to a method for establishing and detecting a circuit board defect detection model. Background Art

[0002] In the production process of electronic products, processes such as motherboard assembly, detection, and packaging are required. However, with the continuous improvement of the performance of electronic products, the increasing complexity of functions, and the characteristics of small volume, high density, and multi-integration presented by the products themselves, the assembly of the motherboard becomes more high-density and multi-integrated. Therefore, the accuracy of the motherboard is inevitable. The current method is traditional manual detection, mainly relying on manual use of simple visual automation detection means. Manual detection is prone to fatigue during long-term fine detection, thus reducing the detection efficiency and accuracy. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for establishing and detecting a circuit board defect detection model that improves detection accuracy.

[0004] To achieve the above purpose, the present invention adopts the following technical solution: A method for establishing a circuit board defect detection model, establishing a defect detection model, where the defect detection model includes a fusion diverse feature extraction module for extracting multi-level features from different levels and a hierarchical attention module for gradually fusing the multi-level feature information extracted from different levels; The fusion diverse feature extraction module includes a convolutional layer for preliminary feature extraction and multiple feature aggregation modules for extracting multi-level features from different levels; The hierarchical attention module includes three fusion layers composed of multiple attention feature fusion blocks. The first fusion layer includes one attention feature fusion block, the second fusion layer includes two attention feature fusion blocks, and the third fusion layer includes three attention feature fusion blocks; The hierarchical attention module further includes multiple iterative feature fusion blocks for iterating the fused features and the output features of the fusion diverse feature extraction module, multiple global average pooling modules, a gated feature fusion block, and a classifier module; One of the global average pooling modules is used to perform a pooling operation on the output of the fusion diverse feature extraction module, and the remaining global average pooling modules are respectively used to perform a pooling operation on the output of each iterative feature fusion block; The gated feature fusion block is used to perform feature fusion on the output of each global average pooling module; The classifier module is used to classify the output of the gated feature fusion block.

[0005] Preferably, the fusion and diversification feature extraction module includes four feature aggregation modules. The output of the convolutional layer serves as the input to the first feature aggregation module, and the output of the first feature aggregation module serves as the input to the next feature aggregation module.

[0006] Preferably, the first fusion layer is used to fuse the preliminary features extracted by the convolutional layer and the output features of the first feature aggregation module. The first attention feature fusion block of the second fusion layer is used to fuse the output features of the first fusion layer and the output features of the second feature aggregation module. The second attention feature fusion block of the second fusion layer is used to fuse the preliminary features extracted by the convolutional layer, the output features of the first attention feature fusion block of the second fusion layer, and the output features of the second feature aggregation module. The first attention feature fusion block of the third fusion layer is used to fuse the output features of the second attention feature fusion block of the second fusion layer and the output features of the third feature aggregation module. The second attention feature fusion block is used to fuse the output features of the first attention feature fusion block of this fusion layer, the output features of the third feature aggregation module, and the output features of the second fusion layer. The third attention feature fusion block fuses the output features of the previous two attention feature fusion blocks of this layer, the output features of the previous two fusion layers, and the preliminary features extracted by the convolutional layer.

[0007] Preferably, the hierarchical attention module includes three iterative feature fusion blocks. The first iterative feature fusion block iterates on the output features of the fourth feature aggregation module and the output features of the first attention feature fusion block of the third fusion layer. The second iterative feature fusion block iterates on the output features of the fourth feature aggregation module, the output features of the second attention feature fusion block of the third fusion layer, and the output features of the first iterative feature fusion block. The third iterative feature fusion block iterates on the output features of the previous two iterative feature fusion blocks, the output features of the third attention feature fusion block of the third fusion layer, and the output features of the fourth feature aggregation module.

[0008] Preferably, the feature aggregation module inputs the feature map into four branches. Among them, two branches respectively perform depthwise separable convolution operations. The feature map output by one of the depthwise separable convolution operations is subjected to average pooling. The other two branches respectively perform upsampling operations. The feature map after one upsampling is directly subjected to Sigmoid activation and then multiplied by the feature map after one depthwise separable convolution. The feature map after the other upsampling passes through convolution, depthwise separable convolution, and Sigmoid activation in sequence and then is multiplied by the feature map after average pooling. The two multiplied feature maps are added together, and then the added feature map is successively subjected to depthwise separable convolution operation, batch normalization operation, and Relu activation operation to obtain the output of the feature aggregation module.

[0009] A circuit board defect detection method includes the following steps: S1: Obtain the design drawing of the sample main board, mark auxiliary division lines on the design drawing, and import them into the drawing database according to the divided areas respectively; S2: Obtain the image of the main board to be tested, preprocess the image of the main board to be tested, design divided areas on the preprocessed image of the main board to be tested according to the main board layout, obtain multiple samples to be tested, input the identified multiple samples to be tested into the corresponding matching database, the matching database retrieves the corresponding design drawing in the drawing database, and obtains the component positions of the samples to be tested after matching. If the component position is an empty area, execute step S3; if the component position is a dense area or an integrated area, execute step S4; S3: Use a single backbone feature extraction network to identify the sample to be tested and obtain the identification result; S4: Use the defect detection model established by the method for establishing any of the above circuit board defect detection models to identify the sample to be tested and obtain the identification result; S5: If the identification result shows that the sample to be tested has a defect, mark the defect position and output the marked image; otherwise, the sample to be tested passes the test.

[0010] Preferably, the preprocessing operation performed on the image of the main board to be tested in step S2 includes image grayscale, image adjustment, image denoising, and image correction performed in sequence.

[0011] By adopting the foregoing design scheme, the beneficial effects of the present invention are as follows: In this application, the image of the main board to be tested is identified by regions, and a single backbone feature extraction network is used for rapid feature vector identification and judgment, avoiding waste of computing power and time by complex detection methods; in the dense area and the integrated area, multi-level features and feature fusion are used for detection, improving the recognition accuracy of the image and ensuring the accuracy of recognition. Description of the Drawings

[0012] Figure 1Schematic diagram of the structure of the defect detection model of the present invention; Figure 2 Processing flowchart of the feature aggregation module of the present invention; Figure 3 Flowchart of the defect detection method of the present invention; Figure 4 Detection flowchart of the defect detection method of the present invention; Figure 5 Schematic diagram of the structure of the single backbone feature extraction network of the present invention. Detailed implementation manners

[0013] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0014] Method for establishing a circuit board defect detection model, establishing a defect detection model, as Figure 1 shown, the defect detection model includes a fusion diverse feature extraction module (DFEM) for extracting multi-level features from different levels and a hierarchical attention module (HAM) for gradually fusing the multi-level feature information extracted from different levels.

[0015] The fusion diverse feature extraction module is used as the basic framework; the incorporation of the hierarchical attention module effectively reduces information loss and greatly enhances the network's ability to capture key features.

[0016] The fusion diverse feature extraction module includes a convolutional layer (Conv) for preliminary feature extraction and multiple feature aggregation blocks (FAB) for extracting multi-level features from different levels; for extracting features from different levels, including local context features, global context features, and spatial gradient features, so as to improve the expression ability of the overall features.

[0017] In this embodiment, the fusion diverse feature extraction module includes four feature aggregation blocks. The output of the convolutional layer serves as the input of the first feature aggregation block, and the output of the first feature aggregation block serves as the input of the next feature aggregation block.

[0018] The hierarchical attention module consists of three fusion layers formed by multiple attention feature fusion blocks (AFFB). The first fusion layer includes one attention feature fusion block, the second fusion layer includes two attention feature fusion blocks, and the third fusion layer includes three attention feature fusion blocks; The hierarchical attention module further includes a plurality of iterative feature fusion blocks (IFFB) for iterating on the fused features and the output features of the fusion diversification feature extraction module, a plurality of global average pooling modules (GAP), a gated feature fusion block (GFFB), and a classifier module; aiming to gradually fuse the multi-level feature information extracted at different levels through the hierarchical attention mechanism, improve the extraction of diverse motherboard features, and enhance the recognition ability of whether there are defects in the motherboard.

[0019] A plurality of attention feature fusion blocks adaptively enhance useful features by introducing an attention mechanism while suppressing useless information; the iterative feature fusion blocks gradually enhance the attention to important features through iterative feature fusion, improving the recognition of anomalies; the gated feature fusion block effectively selects and fuses the most useful features by judging the usefulness of each high-level feature vector.

[0020] One of the global average pooling modules is used to perform a pooling operation on the output of the fusion diversification feature extraction module, and the remaining global average pooling modules are respectively used to perform pooling operations on the outputs of the respective iterative feature fusion blocks one by one; The gated feature fusion block is used to perform feature fusion on the outputs of the respective global average pooling modules; The classifier module is used to classify the output of the gated feature fusion block. In this embodiment, a decision tree classifier and a selection tree classifier can be used as the classifier module.

[0021] In this embodiment, the first fusion layer is used to fuse the preliminary features extracted by the convolutional layer and the output features of the first feature aggregation module; The first attention feature fusion block of the second fusion layer is used to fuse the output features of the first fusion layer and the output features of the second feature aggregation module, and the second attention feature fusion block of the second fusion layer is used to fuse the preliminary features extracted by the convolutional layer, the output features of the first attention feature fusion block of the second fusion layer, and the output features of the second feature aggregation module; The first attention feature fusion block of the third fusion layer is used to fuse the output features of the second attention feature fusion block of the second fusion layer and the output features of the third feature aggregation module, the second attention feature fusion block is used to fuse the output features of the first attention feature fusion block of this fusion layer, the output features of the third feature aggregation module, and the output features of the second fusion layer, and the third attention feature fusion block fuses the output features of the previous two attention feature fusion blocks of this layer, the output features of the previous two fusion layers, and the preliminary features extracted by the convolutional layer.

[0022] The hierarchical attention module includes three iterative feature fusion blocks; The first iterative feature fusion block iterates on the output features of the fourth feature aggregation module and the output features of the first attention feature fusion block of the third fusion layer; The second iterative feature fusion block iterates on the output features of the fourth feature aggregation module, the output features of the second attention feature fusion block of the third fusion layer, and the output features of the first iterative feature fusion block; The third iterative feature fusion block iterates on the output features of the previous two iterative feature fusion blocks, the output features of the third attention feature fusion block of the third fusion layer, and the output features of the fourth feature aggregation module.

[0023] As Figure 2 shown, the feature aggregation module inputs the feature map into four branches. Among them, two branches perform depthwise separable convolution (DWConv) operations respectively. The feature map output by one of the depthwise separable convolution operations is subjected to average pooling. The other two branches perform upsampling operations respectively. One of the upsampled feature maps is directly subjected to Sigmoid activation and then multiplied by the feature map after one of the depthwise separable convolutions. The other upsampled feature map is successively subjected to two-dimensional convolution, depthwise separable convolution, and Sigmoid activation and then multiplied by the feature map after average pooling (AP). The two multiplied feature maps are added together, and then the added feature map is successively subjected to depthwise separable convolution operation, batch normalization operation (BN), and Relu activation function (R) operation to obtain the output of the feature aggregation module. The convolution kernel of the two-dimensional convolution is 3*3, and the dilation factor is 2.

[0024] The operation of the feature aggregation module can be expressed by the following formula: ; ; ; ; ; where is the added feature map, is the depthwise separable convolution operation on the added feature map, is the batch normalization operation on the feature map after the depthwise separable convolution operation, represents the Relu activation function operation, where represents the upsampled feature map, represents the Sigmoid activation function, where () represents the two-dimensional convolution operation with a convolution kernel of 3×3 and a dilation factor of 2.

[0025] By means of bilateral feature fusion, low-level semantic information and high-level detail information are fused to improve the overall anti-interference ability and reduce the probabilities of false detection and missed detection.

[0026] Through global average pooling (GAP), n 1×1 feature maps can be obtained. These 1×1 feature maps are input into the gated feature fusion module (GFFB), and the features are selectively fused through a gating mechanism to improve the fusion effect.

[0027] In this embodiment, a method for detecting a circuit board by using the above defect detection model is also provided.

[0028] The circuit board defect detection method, as Figure 1 shown, includes the following steps: S1: Obtain the design drawing of the sample main board, manually mark the auxiliary dividing lines on the design drawing, and import them into the drawing database according to the divided regions respectively; S2: Obtain the image of the main board to be measured. Use an industrial camera to obtain the image of the main board to be measured on the circuit board production line, preprocess the image of the main board to be measured, divide regions according to the main board layout design on the preprocessed image of the main board to be measured, obtain multiple samples to be measured, input the recognized multiple samples to be measured into the corresponding matching database, the matching database retrieves the corresponding design drawing in the drawing database, and obtain the component positions of the samples to be measured after matching. If the component position is an empty area, execute step S3; if the component position is a dense area or an integrated area, execute step S4; In this embodiment, the preprocessing operation performed on the image of the main board to be measured in step S2 includes image grayscale, image adjustment, image denoising, and image correction performed in sequence.

[0029] Image grayscale: It is a combination of the pixel values of the red channel R, green channel G, and blue channel B. The value range of each channel pixel is from 0 to 255. The purpose of image grayscale is to convert a three-channel color image into a combined channel image according to a preset rule, average the pixel v values of the three RGB channels in the image, calculate the result as the grayscale value, and convert it into a combined channel grayscale image.

[0030] Image adjustment includes various data enhancement techniques such as rotating, flipping, and brightness adjustment of the combined channel grayscale image using conventional techniques.

[0031] Image denoising: Use a Gaussian filter template to scan each row and each column of the image, obtain the new grayscale value of the image through convolution operation, and remove Gaussian noise. Median denoising method: Arrange the grayscale values of the points near the noise points in the image in ascending order to obtain a new sequence, and select the median of the new sequence as the new grayscale value of the noise point.

[0032] Image correction: The Hough transform method and the Hough transform method are used to perform geometric correction on the image. The Hough correction is performed by finding the angle of the maximum overlapping point in the parameters. The Radon algorithm obtains the tilt angle of the image according to the projection angle of the superimposed positioning direction.

[0033] S3: Use a single backbone feature extraction network to identify the sample to be tested and obtain the recognition result; In this embodiment, a conventional ResNet-101 (CNN) as shown in Figure 5 is used as the backbone for feature extraction, and a multi-scale feature fusion module is constructed to aggregate context features and global features. The features are aggregated in a cascaded manner to solve the problem of mismatched feature information at different scales. This model cascades the features of the four stages of the backbone network ResNet-101 through a pyramid pooling module (Pooling), thereby enhancing the ability of feature extraction.

[0034] The operation steps of the single backbone feature extraction network are to perform feature extraction in four stages for ResNet-101. The pyramid pooling module cascades the features extracted in the four stages, performs a convolution operation on the cascaded features, then performs an upsampling operation on the features after the convolution operation respectively, and finally takes the features after the upsampling through a convolution operation as the recognition result for output.

[0035] S4: Use the defect detection model established by the method for establishing any of the above circuit board defect detection models to identify the sample to be tested, as shown in Figure 3 and Figure 4 to obtain the recognition result; S5: If the recognition result shows that the sample to be tested has a defect, mark the defect location and output the marked image. Otherwise, the sample to be tested passes the detection.

[0036] In summary, this application identifies the image of the main board to be tested in regions, uses a single backbone feature extraction network for fast feature vector recognition and judgment, avoiding the waste of computing power and time by complex detection methods; in the dense area and the integrated area, it uses a multi-level feature and feature fusion method for detection, improving the recognition accuracy of the image and ensuring the accuracy of recognition.

[0037] The specific embodiments described above have further detailed the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for establishing a circuit board defect detection model, characterized in that: Establishing a defect detection model, the defect detection model includes a fusion diversified feature extraction module for extracting multi-level features from different levels and a hierarchical attention module for gradually fusing the multi-level feature information extracted from different levels; The fusion diversified feature extraction module includes a convolution layer for preliminary feature extraction and a plurality of feature aggregation modules for extracting multi-level features from different levels; The hierarchical attention module includes three fusion layers consisting of multiple attention feature fusion blocks, the first fusion layer includes one attention feature fusion block, the second fusion layer includes two attention feature fusion blocks, and the third fusion layer includes three attention feature fusion blocks; The hierarchical attention module further includes a plurality of iterative feature fusion blocks for iterating the fused features and the output features of the fused diversified feature extraction module, a plurality of global average pooling modules, a gated feature fusion block and a classifier module; One of the global average pooling modules is used to perform a pooling operation on the output of the fused diversified feature extraction module, and the remaining global average pooling modules are respectively applied one by one to perform a pooling operation on the output of each iterative feature fusion block; The gated feature fusion block is used to perform feature fusion on the outputs of the global average pooling modules; The classifier module is used to classify the output of the gated feature fusion block.

2. The method for establishing a circuit board defect detection model according to claim 1, characterized in that: The fused diversified feature extraction module includes four feature aggregation modules, the output of the convolution layer is used as the input of the first feature aggregation module, and the output of the first feature aggregation module is used as the input of the next feature aggregation module.

3. The method for establishing a circuit board defect detection model according to claim 2, characterized in that: The first fusion layer is used to fuse the preliminary features extracted by the convolution layer and the output features of the first feature aggregation module; The first attention feature fusion block of the second fusion layer is used to fuse the output features of the first fusion layer and the output features of the second feature aggregation module, and the second attention feature fusion block of the second fusion layer is used to fuse the preliminary features extracted by the convolution layer, the output features of the first attention feature fusion block of the second fusion layer, and the output features of the second feature aggregation module; The first attention feature fusion block of the third fusion layer is used to fuse the output features of the second attention feature fusion block of the second fusion layer and the output features of the third feature aggregation module. The second attention feature fusion block is used to fuse the output features of the first attention feature fusion block of the fusion layer, the output features of the third feature aggregation module and the output features of the second fusion layer. The third attention feature fusion block fuses the output features of the first two attention feature fusion blocks of the layer, the output features of the first two fusion layers and the preliminary features extracted by the convolution layer.

4. The method for establishing a circuit board defect detection model as claimed in claim 3, characterized in that: The hierarchical attention module includes three iterative feature fusion blocks; The first iterative feature fusion block iterates the output features of the fourth feature aggregation module and the output features of the first attention feature fusion block of the third fusion layer; The second iterative feature fusion block iterates the output features of the fourth feature aggregation module, the output features of the second attention feature fusion block of the third fusion layer, and the output features of the first iterative feature fusion block; The third iterative feature fusion block iterates the output features of the previous two iterative feature fusion blocks, the output features of the third attention feature fusion block of the third fusion layer, and the output features of the fourth feature aggregation module.

5. The method for establishing a circuit board defect detection model as claimed in claim 4, characterized in that: The feature aggregation module inputs the feature map into four branches, two of which perform depthwise separable convolution operations respectively, and the feature map output by one of the depthwise separable convolution operations is average pooled, and the other two branches perform upsampling operations respectively, one of the upsampled feature maps is directly Sigmoid activated and multiplied with one of the depthwise separable convolution feature maps, and the other upsampled feature map is sequentially convolved, separable depthwise convolution and Sigmoid activated and multiplied with the average pooled feature map, the two multiplied feature maps are added, and then the added feature map is sequentially subjected to depthwise separable convolution operations, batch normalization operations and Relu activation operations to obtain the output of the feature aggregation module.

6. A circuit board defect detection method, characterized in that: The steps include: S1: Obtain a design drawing of a sample motherboard, mark auxiliary dividing lines on the design drawing, and import them into a drawing database according to the divided areas; S2: Acquire the image of the motherboard to be tested, pre-process the image of the motherboard to be tested, divide the area on the pre-processed image of the motherboard to be tested according to the motherboard layout design, obtain multiple samples to be tested, identify the multiple samples to be tested and input them into the corresponding matching database, the matching database retrieves the corresponding design drawings in the drawing database, and obtains the component position of the sample to be tested after matching. If the component position is an open area, execute step S3, and if the component position is a dense area or an integrated area, execute step S4; S3: using a single backbone feature extraction network to identify the sample to be tested and obtain an identification result; S4: using the defect detection model established by the method for establishing a circuit board defect detection model according to any one of claims 1 to 5 to identify the sample to be tested, and obtaining an identification result; S5: If the recognition result shows that the sample to be tested has defects, the defect position is marked and the marked image is output; otherwise, the sample to be tested passes the inspection.

7. The circuit board defect detection method according to claim 6, characterized in that: The preprocessing operations performed on the image of the motherboard to be tested in step S2 include image grayscale, image adjustment, image denoising and image correction performed in sequence.

Images