Method for detecting defect on surface of copper clad laminate on basis of multi-scale gridding

Abstract

The present application provides a method for detecting a defect on the surface of a copper clad laminate on the basis of multi-scale gridding, said method comprising: S1, acquiring and capturing an image of a copper clad laminate by means of a line scan camera; S2, segmenting a copper clad region and a non-copper clad region, and rotating the copper clad region to the front; S3, dividing the copper clad region in the image according to set grid parameters, and performing a pyramid operation on a divided grid graph; S4, calculating the mean of each layer; S5, fusing the means of all of the layers by using a mean degradation coefficient, and obtaining a final background mean; and S6, by using a background-foreground difference, searching for a final defect and classifying same. According to the present application, the amount of computation is reduced, and local information of the background is retained, thus increasing detection speed, and also ensuring the stability of precision. A good detection effect can be obtained when high-speed detection must be performed on a material.

Description

Surface defect detection method of copper clad laminate based on multi-scale gridding

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 5, 2023, with application number 202310817164.2, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to a method for detecting surface defects of copper clad laminates based on multi-scale gridding. Background Art

[0003] As the core material for printed circuit boards (PCBs), copper-clad laminates (CCLs) primarily provide interconnection, conductivity, and insulation, significantly impacting signal transmission within the circuit. Circuit boards made from CCLs are widely used in industries such as communications, healthcare, military, and semiconductors, encompassing virtually all electronic information products. Therefore, as a source material, the quality of CCLs directly impacts the subsequent development of various industries.

[0004] The traditional quality inspection method in the copper clad laminate production line is to locate defects by manually inspecting the material surface. There are many types of defects, such as scratches, dirt, black spots, white spots, bubbles, etc. However, it has the following disadvantages: 1. Due to the properties of the material surface, some subtle defects may not be well perceived by the naked eye. 2. The material is relatively large, and workers need to rotate and flip the material when looking for defects, which can cause secondary damage to the material. 3. Human energy is limited, resulting in defects not being noticed even when they occur. 4. Excessive labor costs 5. All of these will lead to inefficient production line operation.

[0005] Currently, there are related applications of vision in copper-clad laminate (CCL) quality inspection: using visual imaging, the mean of local areas of the CCL within the image is calculated one by one, serving as the background for the board surface. When a defect is present on the board surface, it is detected due to grayscale differences. However, since the mean calculation is affected by the entire area, when the defect's size accounts for a significant portion of the entire local area, the background for the mean calculation will consist almost entirely of the defect's grayscale value, preventing proper defect detection. Furthermore, this method uses a pixel-by-pixel sliding window to calculate the mean, making it very time-consuming when inspecting large targets.

[0006] Summary of the Invention

[0007] The purpose of this application is to provide a method for detecting surface defects of copper clad laminates based on multi-scale gridding. Compared with the previous visual inspection of copper clad laminate quality inspection, the method is faster and more robust for detecting surface defects of copper clad laminates, even if the defects vary in size. To this end, this application adopts the following technical solutions:

[0008] A method for detecting surface defects of copper clad laminates based on multi-scale gridding comprises the following steps:

[0009] S1, collects the image of the copper clad laminate through the line scan camera;

[0010] S2, split the copper-clad area and the non-copper-clad area, and rotate the copper-clad area to the positive direction;

[0011] S3, dividing the copper-clad area of ​​the image according to the set grid parameters, and performing a pyramid operation on the divided grid map; the parameters include: the size of the sliding grid Δx, Δy, the number of pyramid levels N, the pyramid scaling factor S, and the mean degradation coefficient V, wherein the selection of the mean degradation coefficient depends on the number of layers N;

[0012] S4, calculate the mean of each layer;

[0013] S5, through the mean degradation coefficient, fuses the means of each layer to obtain the final background mean;

[0014] S6, finds the final defect and classifies it through background-foreground difference.

[0015] Furthermore, step S2 is specifically as follows:

[0016] S2.1 For each image, separate the copper-clad board and the non-copper-clad area by thresholding; the thresholding is performed by histogram-based foreground and background segmentation;

[0017] After S2.2 segmentation is completed, the copper-clad area is screened out by comparing the outline size. The formula is as follows: c_obj=max(c_obj,c i ),i∈(0,n),

[0018] Where c_obj is the target contour, c i is the selected contour to be compared, n is the total number of contours;

[0019] After S2.3 extracts the outline, a minimum bounding rectangle algorithm is used to determine the actual orientation of the copper clad area, i.e., whether the outline is a "regular" rectangle or a non-regular rectangle, as well as the outline's position within the image. After determining the outline's angle and center position, an affine transformation matrix is ​​established to rotate and translate the image. The copper clad area is now rotated to a "positive" position and subsequently removed from the image for subsequent visual defect detection. (The concept of positive, the definition of angle) "Positive" means that, using the Cartesian coordinate reference system, the angle between one side of the rectangle and the x-axis is 0 degrees.

[0020] Furthermore, the copper clad area is divided into K grids with Δx and Δy as width and height, and a pyramid operation is performed on each grid.

[0021] Number of pyramid layers The number of pyramid layers is limited by the width and height of the grid. When both the width and height exceed 5, the number of layers is forced to 1 and the pyramid operation is stopped. Otherwise, the grid is scaled to N layers using S as the scaling factor.

[0022] The calculation of the mean degradation coefficient V depends on the number of layers N and is the inverse of the number of layers N.

[0023] Furthermore, in step S4, a 4-point interpolation average algorithm is performed on the K*N images, and the formula is as follows:

[0024] In the above formula, p1, p2, p3, and p4 are the four vertices of the grid graph, and f i Represents the pixel values ​​of different vertices, p i x ,p i y Represents the x and y values ​​of the vertex respectively; i = 1, 2, 3, 4; K is the number of grids in each layer, and N is the number of layers;

[0025] Through the above calculations, all grid images and pyramid images have a background value.

[0026] Furthermore, in step S5, all background values ​​are fused to form the final background value of the original area. The background value of each set of pyramid images is extracted, and these background values ​​are multiplied by the degradation coefficient and accumulated to obtain the final background value. The formula is as follows:

[0027] In the above formula, avg is the final background value, a i is the mean of each layer, and V is the mean degradation coefficient.

[0028] Create an empty image of the same size as the original grid image, and fill the entire empty image with avg as the value. Difference the image with the original grid image, set the grayscale tolerance of black and white defects, and detect defects.

[0029] Furthermore, this application uses the deep learning model mobileNetV2 as the migration model;

[0030] The network migration structure based on mobileNetv2 consists of an inverted residual structure, a pyramid structure, an hourglass module and a classification module;

[0031] The construction of the hourglass module relies on three types of sub-modules. Each sub-module is connected through different network layers, namely:

[0032] Submodule 1: dimensionality expansion layer, normalization layer, relu6 activation function layer

[0033] Submodule 2: depth separation layer, normalization layer, relu6 activation function layer

[0034] Submodule 3: dimensionality reduction layer, normalization layer;

[0035] The residual structure refers to the hourglass structure where the input and output features are element-wise added. This is used to fuse features. The classification layer is further connected to two fully connected layers at the end of the network, performing flattening based on the defect type.

[0036] The step S6 is specifically as follows:

[0037] 6.1.1) First, the input image is connected to a convolutional layer in series to start the first layer of feature extraction;

[0038] 6.1.2) The three submodules are combined into an hourglass structure, and different numbers of hourglass structures are connected in series to form an inverted residual structure. This is used to obtain features of different dimensions without excessive parameter usage.

[0039] 6.1.3) A hop connection pyramid operation is performed at the end of the network. This allows for the abstraction of feature information without over-abstraction and loss of information.

[0040] 6.2) Collect defect sample data, label the data by category, and feed the pre-processed data into the network for training. The network will adjust the final output based on the set loss function;

[0041] The loss function used is defined as:

[0042] Where x[cls] is the predicted value, cls is the actual label category, and C is the number of categories. When the loss is smaller, the probability in the brackets is larger, which means that the probability of the sample in the actual category is greater;

[0043] 6.3) After training is complete, the model will be deployed in the actual operating environment. When a defect is detected normally, the defect will be fed into the network for classification, achieving the final defect classification effect.

[0044] When inspecting defects, the visual images obtained are subject to precision requirements, and the amount of data is often quite large. With the solution of this application, since background production no longer requires averaging the entire grid or using a sliding window, the aforementioned technical process reduces the amount of computation while retaining local background information. This not only improves inspection speed but also ensures stable accuracy. This is particularly effective when high-speed inspection of materials is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a flow chart of the detection method of the present application.

[0046] Figure 2 is a histogram that separates the copper-clad area and the non-copper-clad area.

[0047] Figures 3-1 and 3-2 are the pixel step diagram and grid division diagram respectively.

[0048] Figure 4 shows the process of making a grid pyramid.

[0049] FIG5 is a schematic diagram of a 4-point interpolation average algorithm.

[0050] Figure 6 shows the defect detection process, which from left to right are the original image, background image, foreground candidate image and defect image.

[0051] Figure 7 is a diagram showing the composition of the feature pyramid structure.

[0052] Figure 8 is a diagram showing the composition of the hourglass structure module.

[0053] Figure 9 is a diagram showing the composition of the migration model module. DETAILED DESCRIPTION

[0054] The following describes the surface defect detection method of copper clad laminates based on multi-scale gridding in the present application with reference to the accompanying drawings. During the production process, copper clad laminates are continuously conveyed through an assembly line, and the method of the present application performs online real-time detection of defects in copper clad laminates.

[0055] S1. Acquire images using a line scan camera. In this embodiment, four line scan cameras are used to cover the entire surface of the material, capturing each part of the material and acquiring four clear images.

[0056] S2: Divide the copper-covered area and the non-copper-covered area, and rotate the copper-covered area to the positive direction.

[0057] S2.1 For each picture, the copper-clad area and the non-copper-clad area are separated by threshold solution. The threshold solution is based on the histogram-based front and back background segmentation method. The histogram segmentation algorithm counts the pixel values ​​of the entire image from 0 to 255, and the pixel histogram of the background and the pixel histogram of the copper-clad area will show a double peak phenomenon. As shown in Figure 2, for the histogram, the noise value is first filtered by a fixed threshold T, and the pixel values ​​less than T are suppressed. Then the histogram is calculated twice for the maximum value, and the position representing the background and the copper-clad area is found. The average of the two is used as the final segmentation threshold for separation of the copper-clad area and the background. The following is the histogram solution formula:

[0058] Where p(r i ) represents the histogram, n i Represents grayscale r i MN is the total number of pixels. In Figure 2, the horizontal axis is the pixel value, and the vertical axis is the number of pixels in the image at the current pixel value. Assuming a is the background peak and b is the copper-clad peak, the mean between the two, t = (a + b) / 2, is used as the final segmentation threshold. Values ​​below the threshold are considered background, and values ​​above the threshold are considered copper-clad.

[0059] After the segmentation in S2.2 is completed, contour extraction is required. At this time, the extracted contour may contain not only copper-clad areas but also noise areas. Since copper-clad areas are large targets in the image, the copper-clad areas are screened out by comparing the contour sizes. The formula is as follows: c_obj=max(c_obj,c i ),i∈(0,n),

[0060] Where c_obj is the target contour, c i is the selected contour to be compared, and n is the total number of contours.

[0061] After S2.3 extracts the contour, it is necessary to determine the actual orientation of the copper clad area, that is, whether the contour is a "regular" rectangle or a non-"regular" rectangle, and the position of the contour in the image. The minimum bounding rectangle algorithm can solve the target problem very well. After obtaining the contour angle and center position, the image is rotated and translated by establishing an affine transformation matrix, and the copper clad area will be rotated to the "regular" position. The copper clad area will then be cut out from the image for subsequent visual defect detection. The following is the formula for establishing the affine transformation matrix:

[0062] In the matrix, θ is the angle of the screened copper clad area, tx, ty represent the relative amount of center translation required, x, y represent the coordinate points to be transformed, and x′, y′ represent the coordinates of the point after rotation and translation.

[0063] After completing the above operations, the copper cladding area map is obtained.

[0064] S3: Set the grid size Δx, Δy, the number of pyramid levels N, the pyramid scaling factor S, and the mean degradation coefficient V, where the selection of the degradation coefficient depends on the number of levels N.

[0065] The length and width of the grid are both Δx and Δy. Δx <W c &&Δy <H c , where W c ,H c The entire copper-clad area is thus divided into multiple grids. Traditional background production requires acquiring image blocks pixel by pixel using a sliding window and calculating the overall mean of the image blocks as the background value. This is time-consuming and easily affected by the grayscale and size of internal defects. The method proposed in this application avoids the above problems.

[0066] Number of pyramid layers The number of pyramid layers is limited by the width and height of the grid. When both the width and height exceed 5, the number of layers is forced to 1 and the pyramid operation is stopped.

[0067] The pyramid coefficient is used to establish the width and height of each layer of the pyramid image, that is, W N =W*S,H N =H*S, where W N H is the width of the grid after scaling to the Nth level. N The height of the grid after scaling to level N.

[0068] The calculation of the mean degradation coefficient V depends on the number of layers N and is the inverse of the number of layers N.

[0069] Gridding the copper area: Using the set grid width and height, move along the X-axis of the copper area at a grid width and step size of Δx, and along the Y-axis at a grid height and step size of Δy, to obtain the coordinates of the four corner points of the grid. In this way, the copper area is divided into K parts, that is, K grid maps.

[0070] After completing the above operations, the obtained grid image is scaled to pyramid size. For each grid image, it is scaled to N images. The width and height of each image are calculated as follows: i＝0,1,2...N =W0*S*(i+1) H i＝0,1,2...N =H0*S*(i+1)

[0071] Where W0, H0 are the width and height of the first grid. It is worth noting that when W i ,H iWhen the pixel value is less than 5, the pyramid behavior stops. A series of pyramid images created from each grid image is called a group. K grid images, that is, K groups of pyramid images, totaling K*N images. The process is shown in Figure 4.

[0072] S4: Calculate the mean of each K*N graph generated above. Based on the grid-based 4-point interpolation average algorithm. Take one of the graphs as an example, as shown in Figure 5, take p1, p2, p 3 ,The pixel grayscale values ​​of the four vertices of p4 are interpolated through bilinear interpolation to get the grayscale value of the center point of the p5 grid, and this grayscale value is used as the background value of the current grid.

[0073] The interpolation formula is:

[0074] In the above formula, f i Represented as pixel values ​​of different vertices of the grid graph, p i x ,p i y Represents the x and y values ​​of the vertex respectively; i=1,2,3,4.

[0075] Through the above calculations, all grid images and pyramid images have a background value.

[0076] S5: Fuse all background values ​​to form the final background value of the original area. Extract the background value of each set of pyramid images, multiply these background values ​​by the degradation coefficient, and accumulate them. The final background value is obtained. The formula is as follows:

[0077] In the above formula, avg is the final background value, a i is the mean value of each layer, and V is the degradation coefficient.

[0078] S6: Create an empty image of the same size as the original grid image and fill it with the value avg. Perform pixel-level differences between this image and the original grid image, and set grayscale tolerances for black and white defects. Defects within the tolerance range are considered defects. The detection results are shown in Figure 6. After defects are detected, they need to be classified.

[0079] Traditional classification is done through manual feature extraction, such as the aspect ratio, roundness, concavity, etc. of the defect. These attributes are used as features and sent to the classifier for classification training. However, due to the limitations of the feature attributes, the classification accuracy, robustness and scalability are poor, especially when new defects appear in actual working conditions, the parameters cannot be adjusted well, resulting in the destruction of the previously set classification conditions. Taking into account the above problems and the real-time problems of actual on-site detection, this application uses the deep learning model mobileNetV2 as a migration model. Continue to chain the fully connected layer based on the original network, and redesign the classification layer according to the type of defect. The details are as follows:

[0080] The network migration structure based on mobileNetv2 consists of an inverted residual structure, a feature pyramid structure (different from the aforementioned pyramid, see Figure 7), an hourglass module (see Figure 8) and a classification module.

[0081] The construction of the hourglass module relies on three types of sub-modules. Each sub-module is connected through different network layers. They are:

[0082] Submodule 1: dimensionality expansion layer, normalization layer, relu6 activation function layer

[0083] Submodule 2: depth separation layer, normalization layer, relu6 activation function layer

[0084] Submodule 3: dimensionality reduction layer, normalization layer.

[0085] The residual structure refers to the hourglass structure where the input and output features are element-wise added. This is used to fuse features. The classification layer is connected to two fully connected layers at the end of the network, performing flattening based on the defect type. The detailed steps are as follows:

[0086] 6.1.1) First, the input image is connected in series with a convolutional layer to start the first layer of feature extraction.

[0087] 6.1.2) The three submodules are combined into an hourglass structure, and different numbers of hourglass structures are connected in series to form an inverted residual structure. This is used to obtain features of different dimensions without excessive parameter usage.

[0088] 6.1.3) A skip-connection pyramid operation is performed at the end of the network, which allows for the abstraction of feature information without over-abstraction and loss of information.

[0089] The network structure is shown in Figure 9. In Figure 9, the English meanings are:

[0090] Conv2d: convolution layer; Element_wise_add: element-wise overlay layer; Avg_layer: pooling layer; Fc layer: fully connected layer; Flatten layer: classification layer.

[0091] 6.2) Collect defect sample data and label the data by category. Send the pre-processed data to the network for training. The network will adjust the final output according to the set loss function.

[0092] The loss function used is defined as:

[0093] Where x[cls] is the predicted value, cls is the true label category, and C is the number of categories. When the loss is smaller, the probability in the brackets is larger, which means that the probability of the sample being in the true category is greater.

[0094] 6.3) After training is complete, the model will be deployed in the actual operating environment. When a defect is detected normally, the defect will be fed into the network for classification, achieving the final defect classification effect.

Claims

1. A method for detecting surface defects of a copper clad laminate based on multi-scale gridding, comprising the following steps: S1, collect the image of the copper clad laminate through the line scan camera; S2, divide the copper-clad area and the non-copper-clad area, and rotate the copper-clad area to the positive direction; S3, dividing the copper-clad area of ​​the image according to the set grid parameters, and performing a pyramid operation on the divided grid map; the parameters include: the size of the sliding grid Δx, Δy, the number of pyramid levels N, the pyramid scaling coefficient S, and the mean degradation coefficient V, wherein the selection of the mean degradation coefficient depends on the number of layers N; S4, calculate the mean of each layer; S5, through the mean degradation coefficient, fuses the means of each layer to obtain the final background mean; S6, finds and classifies the final defects through background-foreground differences.

2. A method for detecting surface defects of copper clad laminates based on multi-scale gridding as claimed in claim 1, wherein: Step S2 is specifically as follows: S2.1 For each image, the copper-clad board and the non-copper-clad area are separated by threshold solution; the threshold solution is based on the histogram-based foreground and background segmentation method; After S2.2 segmentation is completed, the copper-clad area is screened out by comparing the contour size. The formula is as follows: c_obj=max(c_obj,c i ),i∈(0,n), Where c_obj is the target contour, c i is the selected contour to be compared, n is the total number of contours; S2.3 After the contour is extracted, the minimum circumscribed rectangle algorithm is used to determine the actual orientation of the copper-clad area, that is, whether the contour is a "regular" rectangle or a non-"regular" rectangle, and the position of the contour in the image; after obtaining the contour angle and center position, the image is rotated and translated by establishing an affine transformation matrix. The copper-clad area will be rotated to the "positive" position at this time, and then the copper-clad area will be cut out from the image for subsequent visual defect detection.

3. A method for detecting surface defects of copper clad laminates based on multi-scale gridding as claimed in claim 1, wherein: Divide the copper-clad area into K grids with Δx and Δy as width and height, and perform pyramid operation on each grid. Pyramid Layers The number of pyramid layers is limited by the width and height of the grid map. When both the width and height exceed 5, the number of layers is forced to be set to 1 and the pyramid operation is stopped. Otherwise, the grid map is scaled to N layers with S as the scaling factor. The calculation of the mean degradation coefficient V depends on the number of layers N and is the inverse of the number of layers N.

4. A method for detecting surface defects of copper-clad laminates based on multi-scale gridding according to claim 1, wherein in step S4, a 4-point interpolation average algorithm is performed on K*N pictures, and the formula is as follows: In the above formula, p1, p2, p3, p4 are the four vertices of the grid graph, and f i Represents the pixel values ​​of different vertices, Represents the x and y values ​​of the vertex respectively; i = 1, 2, 3, 4; K is the number of grids in each layer, and N is the number of layers; Through the above calculation, all grid images and pyramid images have a background value.

5. A method for detecting surface defects of copper clad laminates based on multi-scale gridding as claimed in claim 1, wherein: In step S5, all background values ​​are fused as the final background value of the original area, and the background value of each group of pyramid images is extracted. These background values ​​are multiplied by the degradation coefficient and accumulated to obtain the final background value; the formula is as follows: In the above formula, avg is the final background value, a i is the mean of each layer, and V is the mean degradation coefficient. Create an empty image of the same size as the original grid image, and fill the entire empty image with avg as the value. Difference the image with the original grid image, set the grayscale tolerance of black and white defects, and detect defects.

6. A copper clad laminate surface defect detection method based on multi-scale gridding as claimed in claim 1, using a deep learning model mobileNetV2 as a migration model; The network migration structure based on mobileNetv2 consists of an inverted residual structure, a pyramid structure, an hourglass module and a classification module; The construction of the hourglass module depends on three types of sub-modules. Each sub-module is connected through different network layers, namely: Submodule 1: dimensionality expansion layer, normalization layer, relu6 activation function layer Submodule 2: depth separation layer, normalization layer, relu6 activation function layer Submodule 3: dimensionality reduction layer, normalization layer; The residual structure refers to the element wise add operation of the input features and output features of the hourglass structure. It is used to fuse the features. The classification layer continues to connect two fully connected layers at the end of the network and performs flatten processing according to the defect type; The step S6 is specifically as follows: 6.1.1) First, the input image is connected in series with a convolutional layer to start the first layer of feature extraction; 6.1.2) The three submodules form an hourglass structure, and different numbers of hourglass structures are connected in series to form an inverted residual structure. This is used to obtain features of different dimensions without excessive number of parameters. 6.1.3) A skip connection pyramid operation is performed at the end of the network. This allows the feature information to be abstracted while ensuring that there is no loss of information due to excessive abstraction; 6.2) Collect defect sample data, label the data by category, and send the pre-processed data to the network for training. The network will adjust the final output according to the set loss function; The loss function used is defined as: Where x[cls] is the predicted value, cls is the actual label category, and C is the number of categories. When the loss is smaller, the probability in the bracket is larger, which means that the probability of the sample in the actual category is greater; 6.3) After training is completed, the model will be deployed in the actual operating environment. When a defect is detected normally, the defect will be sent to the network for classification. The final defect classification effect is achieved.

Images