A product defect detection method based on parallel multi-scale fusion network

Abstract

The present invention discloses a product defect detection method based on a parallel multi-scale fusion network, comprising: inputting an industrial product image to be detected into a product defect detection network model obtained through training, and detecting and segmenting defect areas in the industrial product image. The product defect detection network model is obtained through training, comprising: constructing a product defect detection network model; obtaining a training data set, the training data set comprising a historical industrial product image and a segmentation mask of a corresponding defect area; sending the training data set into the product defect detection model, constructing a mapping relationship between the historical industrial product image and the segmentation mask of the corresponding defect area; and according to an aggregate loss function, using an Adam optimizer to implement iterative optimization and updating of product defect detection network model parameters. The present invention realizes the segmentation of defect areas in industrial product images and improves the accuracy of segmentation.

Description

Technical Field

[0001] The invention relates to a product defect detection method based on a parallel multi-scale fusion network, and belongs to the technical field of image processing. Background Art

[0002] Surface defect detection is an indispensable process in the industrial production process and a very important research topic in computer vision. In the production process, due to the complex production process and production environment, surface defects are inevitable. These defects not only affect the appearance of the product, but also damage the life and safety of the product. Therefore, product defect detection is an indispensable inspection procedure. In previous industrial production, relying on traditional manual visual inspection methods, there are problems such as low efficiency, high cost, poor safety and leakage rate. In recent years, with the development of machine learning and deep learning, surface defect detection based on computer vision has been widely used in 3C, semiconductor, electronics, pharmaceutical, chemical and machinery manufacturing industries, which can effectively solve the problem of relying on manual inspection. At present, automatic detection methods based on machine vision can solve existing manual inspection problems such as low efficiency, high labor intensity and subjective factors, which has attracted the interest of more and more researchers. However, due to the noise, human interference, diversity and complexity of defect types in the production environment, the accuracy of existing industrial product surface defect segmentation and positioning methods still cannot meet the actual application needs. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a product defect detection method based on a parallel multi-scale fusion network.

[0004] To achieve the above-mentioned purpose, the present invention provides a product defect detection method based on a parallel multi-scale fusion network, comprising: inputting an industrial product image to be detected into a trained product defect detection network model, and detecting and segmenting to obtain defect areas in the industrial product image.

[0005] Prioritize, train and obtain a product defect detection network model, including:

[0006] Build a product defect detection network model;

[0007] Obtain a training data set, the training data set including historical industrial product images and corresponding segmentation masks of defective areas;

[0008] According to the aggregate loss function, the training data set is fed into the product defect detection model to build a mapping relationship between the historical industrial product images and the corresponding segmentation masks of the defect areas. The Adam optimizer is used to iteratively optimize and update the parameters of the product defect detection network model.

[0009] If the aggregate loss function converges to a certain value or reaches the set number of iterations, the training ends and the final product defect detection network model is obtained.

[0010] Preferably, the constructed product defect detection network model includes a backbone network, a parallel multi-scale channel and a fusion detection module, and the backbone network, the parallel multi-scale channel and the fusion detection module are connected in sequence;

[0011] The backbone network performs multiple parallel multi-scale feature extractions.

[0012] Preferably, the aggregate loss function is:

[0013]

[0014] Where N is the batch size, k is the number of deep supervision signals, and μ k represents the weight of the k-th side output fusion loss, L bce represents the binary cross entropy loss, L ssim represents the SSIM loss, L iou represents the boundary IOU loss, and θ is the training hyperparameter of the product defect detection network model to be iteratively optimized and updated.

[0015] Primarily, the binary cross entropy loss is calculated as:

[0016]

[0017] Where G(x,y)∈{0,1} is the true label of the real defect area, Y(x,y) is the probability of the defective area predicted by the product defect detection network model, and (x,y) is the pixel coordinate of the historical industrial product image;

[0018] The calculation formula of SSIM loss is:

[0019]

[0020] In the formula, μ x is the mean of x, μ y is the mean of y, σ x is the standard deviation of x, σ y is the standard deviation of y, σ xy is the covariance of x and y, C1 and C2 are set constants;

[0021] The calculation formula for boundary IOU loss is:

[0022]

[0023] Where H is the height of the historical industrial product image, and W is the width of the historical industrial product image.

[0024] Prioritize, according to the aggregate loss function, the training data set is fed into the product defect detection model, the mapping relationship between the historical industrial product images and the corresponding defect areas is constructed, and the Adam optimizer is used to implement the iterative optimization and update of the parameters of the product defect detection network model, including:

[0025] Through the back propagation algorithm, the network parameters ω of the product defect detection network model are updated, ω←ω+α·Adam(ω,d ω ), where Adam is the gradient descent algorithm, α is the learning rate, d ω is the gradient update of the network parameter ω.

[0026] Preferably, the parallel multi-scale pipeline includes a dual threshold shrinkage module and a contextual attention module,

[0027] The parallel multi-scale channel expression is:

[0028] y n =C[D(D(X n )+X n )],

[0029] In the formula, y n is the n-th scale feature map output by the parallel multi-scale channels, D(·) is the dual threshold shrinkage module, and C(·) is the contextual attention module.

[0030] Preferably, the fusion detection module expression is:

[0031]

[0032] Where σ is the sigmoid activation function, and Y is the defect area predicted by the product defect detection network model.

[0033] Preferably, the expression of the dual threshold shrinkage module is:

[0034]

[0035] Where X is the parallel multi-scale feature, X = {X n :n=0,1,2……5}, T is the dual threshold corresponding to each parallel multi-scale feature, (c,i,j) represents the channel, height and width of each parallel multi-scale feature X respectively;

[0036] The expression of the contextual attention module is:

[0037]

[0038] Where, S = {S n:n=0,1,2……5} is the parallel multi-scale feature after two rounds of dual threshold shrinkage module processing in the parallel multi-scale channel, y is the parallel multi-scale feature output by the context attention module, y={y n :n=0,1,2……5};g h , g w are the attention weights in the horizontal and vertical directions of parallel multi-scale features, respectively.

[0039] The beneficial effects achieved by the present invention are:

[0040] The present invention proposes a product defect detection based on a parallel multi-scale fusion network. The constructed product defect detection network model uses a designed dual threshold shrinkage module and a context attention module to connect in series to form a parallel multi-scale channel module for multi-scale feature processing, relies on a fusion detection module to fuse multi-scale features, and outputs the final anomaly detection result to achieve effective boundary prediction; the present invention realizes the segmentation of defect areas of industrial product images and improves the accuracy of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the present invention;

[0042] Figure 2 It is a structural schematic diagram of the product defect detection network model in the present invention;

[0043] Figure 3 The schematic diagram of the structure of the dual threshold shrinkage module in the present invention is

[0044] Figure 4 Schematic diagram of the structure of the context attention module in the present invention.

[0045] Figure 5 This is a schematic diagram of the visualized comparison results of the processing of the Northeastern University hot-rolled strip surface defect dataset by the present invention. DETAILED DESCRIPTION

[0046] The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and are not intended to limit the protection scope of the present invention.

[0047] Embodiment 1

[0048] The present invention inputs the industrial product image data to be segmented into a product defect detection network model obtained through training, realizes the detection of industrial product image defects, and segments the defective area of ​​the industrial product image.

[0049] Specifically, the technical solution adopted by the present invention is:

[0050] A product defect detection based on a parallel multi-scale fusion network includes the following steps:

[0051] S101, collecting industrial product images, marking the segmentation masks contained in the defect segmentation areas on the industrial product images, and using them as training data sets;

[0052] S201, constructing a product defect detection network model based on a parallel multi-scale fusion network, wherein the product defect detection network model consists of a backbone network, a parallel multi-scale channel and a fusion detection module;

[0053] S301, designing an aggregate loss function for training and learning the product defect detection network model, and presetting training hyperparameters involved in the product defect detection network model;

[0054] S401, sending the training data set into the product defect detection model, and using the Adam optimizer to iteratively optimize and update the parameters of the product defect detection network model according to the aggregate loss function, and the training is completed;

[0055] S501. Input the industrial product image into the trained product defect detection network model to realize automatic detection and segmentation of defect areas in the industrial product image.

[0056] 1. In step S101, annotate industrial product images with different levels of noise at the pixel level using annotation software to obtain segmentation masks of defective areas.

[0057] 2. In step S201, a product defect detection network model based on a parallel multi-scale fusion network is constructed. The product defect detection network model mainly consists of a backbone network, a parallel multi-scale channel and a fusion detection module. First, the backbone network is used to extract multiple parallel multi-scale features. The nth level feature of the parallel multi-scale feature extraction output of the backbone network is Among them C i represents the number of channels corresponding to parallel multi-scale features, H n , W n Represents the height and width of the parallel multi-scale feature, where n = 0, 1, 2, ..., 5, and there are six parallel multi-scale features in total.

[0058] The parallel multi-scale channel module processes the extracted parallel multi-scale features, sends these six parallel multi-scale features to the fusion detection module for multi-scale feature fusion, and outputs the final defect area.

[0059] 3. In step S301, the aggregation loss function is:

[0060]

[0061] Where N is the batch size; k is the number of deep supervision signals, which is 8 according to the side output signal.k L represents the weight of the k-th side output fusion loss, which is empirically set to 1. bce represents the binary cross entropy loss, L ssim represents the SSIM loss, L iou represents the boundary IOU loss, ω is the training hyperparameter of the product defect detection network model that needs to be iteratively optimized and updated, and the training hyperparameter of the product defect detection network model includes the model learning rate α and the number of iterations epoch. The learning of the training hyperparameter is guided by minimizing the loss function (1).

[0062] Furthermore, we define the binary cross entropy loss to constrain the difference in probability distribution between the network prediction value and the true label. It is defined as:

[0063]

[0064] Where G(x,y)∈{0,1} is the segmentation mask of the real defect area, and Y(x,y) is the probability of the defect area predicted by the network model.

[0065] Furthermore, SSIM loss was originally used for image quality assessment. It captures the structural information in the defect segmentation image. Therefore, it is integrated into the aggregation loss to guide the network to learn the structural information about the defective object from the true defect segmentation mask. Let Y = {Y j :j=1,…,N 2} and G={G j :j=1,…,N 2} is the pixel value of two corresponding N×N size features of the model predicted defect area probability Y and the true defect segmentation mask G, and the SSIM loss is defined as:

[0066]

[0067] In the formula, μ x , μ y and σ x , σ y are the mean and standard deviation of x and y respectively, σ xy is the covariance of x and y, C1 = 0.012, C2 = 0.032. Further, IOU loss was originally used to measure the similarity between two groups, and then used as a standard evaluation metric for object detection and segmentation. IOU loss can be expressed as:

[0068]

[0069] Where G(x,y)∈{0,1} is the segmentation mask of the real defect area, Y(x,y) is the defect area probability predicted by the network model, H is the image height, and W is the image width.

[0070] 4. Step S401 includes:

[0071] Step S4011, initializing the corresponding parameters of each network module; inputting a training data set, and constructing a mapping relationship between a training industrial product image and a corresponding real defect area segmentation mask;

[0072] Step S4012: Update the network parameters ω of each neural network model through the back propagation algorithm, ω←ω+α·Adam(ω,d ω ), where Adam is a gradient descent algorithm, α is the learning rate, set to 0.001, d ω is the gradient update of the network model parameter ω;

[0073] Step S4013: For all images in the entire training data set, perform steps S401 and S402 in sequence, and perform a total of epoch=600 iterations or the aggregate loss function converges to a certain value, then the training ends.

[0074] 5. Step S501 includes: inputting the test image into the product defect detection network model to achieve defect segmentation of the industrial product image.

[0075] First, the backbone network extracts multi-scale features from the test image. The multi-scale feature expression is:

[0076] X n =F(I)#(5)

[0077] In the formula, I is the test image, X n Represents the n-th level feature of the parallel multi-scale feature extraction output of the backbone network F(·), n = 0, 1, 2...5, and there are six parallel multi-scale features in total.

[0078] Then each parallel multi-scale feature X = {X n :n=0,1,2……5} The forward input is a parallel multi-scale channel, which consists of a dual threshold shrinkage module and a context attention module. The output expression is:

[0079] y n =C[D(D(X n )+X n )]#(6)

[0080] Where D(·) is the dual threshold shrinkage module, C(·) is the context attention module, and X n Represents the feature map of the nth scale, n = 0, 1, 2...5, y n It is the nth scale feature map output by the parallel multi-scale channel.

[0081] Finally, the scale features of the parallel multi-scale channel module are input into the fusion detection module, which outputs the final defect detection result by fusing the scale features into the context attention module. The output expression of the fusion detection module is:

[0082]

[0083] Where σ is the sigmoid activation function, and Y is the final defect segmentation prediction map.

[0084] Furthermore, the dual threshold shrinkage module is Figure 3 As shown, the parallel multi-scale feature X = {X n :n=0,1,2……5} forward input dual threshold shrinkage module, the output expression of dual threshold shrinkage module is:

[0085]

[0086] In the formula, T is the dual threshold corresponding to each scale feature, and (c, i, j) represents the three dimensions of channel, height and width of each scale feature X. The dual threshold performs threshold shrinkage processing on each scale feature X in the horizontal and vertical directions. The sum of the dual thresholds can be expressed as:

[0087] T(c,i,j)=γ h (c,j)×X h (c,j)+γ v (c,i)×X v (c,i)#(9)

[0088] In the formula, × represents the element-by-element multiplication between features, X h , X v Pooling operations are performed along the horizontal and vertical directions respectively, and after absolute value and average operations, the obtained parallel multi-scale features X in two directions are two encoding vectors in the horizontal and vertical directions of the absolute value features. h and γ v are the scaling parameters in the horizontal and vertical directions respectively. The encoding output in the horizontal direction can be expressed as:

[0089]

[0090] Similarly, the encoding output in the vertical direction can be written as:

[0091]

[0092] Then, the encoding vector X h , X vAfter two 1×1 convolution operations F respectively, and after Sigmoid activation function processing, its value range is constrained to (0,1), and the scaling parameter γ is obtained. h and γ v The calculation process can be expressed as:

[0093]

[0094] Similarly, the vertical output can also be expressed as:

[0095]

[0096] Further, the scaling parameter γ h and γ v Multiply by X h , X v , we can get the thresholds of the two dimensions. This operation can keep the thresholds within a reasonable range. In formula (9), during the element-by-element addition of the two thresholds, a Reweight operation is performed, that is, the horizontal dimension γ h (c,j)×x h The dimension of the weight obtained element by element in (c, j) is expanded from C×H×1 to C×H×W, and the vertical dimension is also reweighted to achieve threshold shrinkage denoising in the entire feature space.

[0097] Furthermore, the context attention module is Figure 4 As shown in the figure, the parallel multi-scale feature S = {S i :i=0,1,2……5}, continue forward through the context attention module, the output of the context attention module C(·) can be written as:

[0098]

[0099] y is the multi-scale output feature of the context attention module in the parallel multi-scale channel, y = {y n :n=0,1,2……5}. g h , g w are the attention weights in the horizontal and vertical directions respectively. The two attention weights are obtained by first using two pooling kernels (H, 1) and (1, W) to encode each channel along the horizontal and vertical coordinates respectively. Therefore, the encoding output of the cth channel of the multi-scale feature map with a height of h can be expressed as:

[0100]

[0101] Similarly, the encoding output of the cth channel of the multi-scale feature map with width w can also be written as:

[0102]

[0103] The above two transformations aggregate features with two spatial directions respectively and generate a pair of direction-aware feature maps. These two transformations allow the attention block to capture position dependencies along one spatial direction and preserve precise position information with another spatial direction, which helps the network to locate defective objects more accurately.

[0104] Next, the same shared 1×1 convolution operation F is used along both directions to obtain:

[0105]

[0106] Where δ is the Sigmoid activation function, f h ∈R C / r×1×w and f w ∈R C / r×h×1 is an intermediate feature map that encodes spatial information in the horizontal and vertical directions. Here, r is the scaling factor that controls the size of the feature map channels. Finally, using another two 1×1 convolution operations F h and F w f h and f w Improve the channel and get the attention weights in two directions:

[0107] g h =σ(F h (f h ))#(18)

[0108] g w =σ(F w (f w ))#(19)

[0109] In the formula, σ is the sigmoid function, which is used to realize the channel fusion process. Then use the Reweight operation to expand the output g h , g w , respectively used as attention weights. After the parallel multi-scale channel, the scale features in the output y processed by the context attention module are input into the fusion detection module, and the final defect detection result is output by fusing the scale features input into the context attention module, that is, the final defect segmentation prediction map Y.

[0110] Embodiment 2

[0111] In order to verify the effect of the present invention and the effectiveness of the proposed dual threshold shrinkage module and context attention module, simulation experiments and ablation experiments were conducted on the present invention. The test column size is 256×256, the Northeastern University hot-rolled strip surface defect data set is used to train and test the model, and the relevant parameters are set: α=0.001, epoch=600. The experimental evaluation uses a quantitative analysis method. By conducting experiments on the test set, the final segmentation accuracy is MAE=0.0106, Weighted F-measure=0.9344, and Structure measure=0.9462.

[0112] Ablation experiments are also conducted on the test set to verify the effectiveness of the dual threshold shrinkage module and the contextual attention module. By removing the dual threshold shrinkage module (DDM) and the contextual attention module (CAM) respectively, only the ResNet-34-CBAM network model is retained for comparison with the complete network structure. The ablation experiment results are shown in Table 1. The final network output results are shown in Figure 5 As shown, under the condition of high noise and complex background, the defect positioning result of the present invention is the best.

[0113] It can be seen from Table 1 that compared with the classification network that only retains the ResNet-34-CBAM model, the dual threshold shrinkage module and the context attention module can indeed effectively improve the defect segmentation accuracy as described above.

[0114] Table 1

[0115]

[0116]

[0117] The present invention also shows the quantitative index comparison with other network models on the Northeastern University hot-rolled strip surface defect dataset. The evaluation indicators include MAE, WF (Weighted F-measure), and SM (Structure measure), as shown in Table 2:

[0118] Table 2

[0119]

[0120] As can be seen from Table 2, the model of this patent can effectively segment product defects under three different noise levels, and the patent has achieved greater performance improvement on the data set in terms of indicators. Among them, compared with the BASNet method under ρ = 0, WF and SM are improved by 29.8% and 2.0% on average, and the MAE metric is significantly reduced by 30.2%. Under other noise levels, the model of this patent has also achieved great improvements in various indicators.

[0121] from Figure 5 It can be seen that the network model of this patent can effectively detect surface defects of products in complex scenarios, regardless of the overall structure of the defects or the fineness of the defect boundaries.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A product defect detection method based on a parallel multi-scale fusion network, characterized in that: include: Inputting the industrial product image to be detected into the trained product defect detection network model, detecting and segmenting to obtain the defect area in the industrial product image; Among them, the product defect detection network model includes a backbone network, a parallel multi-scale channel and a fusion detection module, and the backbone network, the parallel multi-scale channel and the fusion detection module are connected in sequence; The parallel multi-scale pipeline includes a dual threshold shrinkage module and a contextual attention module. The parallel multi-scale channel expression is: y n =C[D(D(X n )+X n )] In the formula, y n is the n-th scale feature map output by the parallel multi-scale channel, D(·) is the dual threshold shrinkage module, and C(·) is the context attention module; The expression of the dual threshold shrinkage module is: Where X is the parallel multi-scale feature, X = {X n : n = 0, 1, 2...5}, T is the dual threshold corresponding to each parallel multi-scale feature, (c, i, j) represents the three dimensions of channel, height and width of each parallel multi-scale feature X respectively; The expression of the contextual attention module is: In the formula, S={S n :n=0,1,2......5} is the parallel multi-scale feature after two rounds of dual threshold shrinkage module processing in the parallel multi-scale channel, y is the parallel multi-scale feature output by the context attention module, y={y n :n=0,1,2......5};g h , g w are the attention weights in the horizontal and vertical directions of parallel multi-scale features, respectively.

2. The product defect detection method based on parallel multi-scale fusion network according to claim 1 is characterized in that: Train and obtain the product defect detection network model, including: Build a product defect detection network model; Obtain a training data set, the training data set including historical industrial product images and corresponding segmentation masks of defective areas; According to the aggregate loss function, the training data set is fed into the product defect detection model to build a mapping relationship between the historical industrial product images and the corresponding segmentation masks of the defect areas. The Adam optimizer is used to iteratively optimize and update the parameters of the product defect detection network model. If the aggregate loss function converges to a certain value or reaches the set number of iterations, the training ends and the final product defect detection network model is obtained.

3. The product defect detection method based on parallel multi-scale fusion network according to claim 2 is characterized in that: The aggregate loss function is: Where N is the batch size, k is the number of deep supervision signals, and μ k represents the weight of the k-th side output fusion loss, L bce represents the binary cross entropy loss, L ssim represents the SSIM loss, L iou represents the boundary IOU loss, and θ is the training hyperparameter of the product defect detection network model to be iteratively optimized and updated.

4. The product defect detection method based on parallel multi-scale fusion network according to claim 3 is characterized in that: The binary cross entropy loss is calculated as: Where G(x, y)∈{0,1} is the true label of the real defect area, Y(x, y) is the probability of the defective area predicted by the product defect detection network model, and (x, y) is the pixel coordinate of the historical industrial product image; The calculation formula of SSIM loss is: In the formula, μ x is the mean of x, μ y is the mean of y, σ x is the standard deviation of x, σ y is the standard deviation of y, σ xy is the covariance of x and y, C1 and C2 are set constants; The calculation formula for boundary IOU loss is: Where H is the height of the historical industrial product image, and W is the width of the historical industrial product image.

5. The product defect detection method based on parallel multi-scale fusion network according to claim 2 is characterized in that: According to the aggregate loss function, the training data set is fed into the product defect detection model to build a mapping relationship between historical industrial product images and corresponding defect areas. The Adam optimizer is used to iteratively optimize and update the parameters of the product defect detection network model, including: Through the back propagation algorithm, the network parameters ω of the product defect detection network model are updated, ω←ω+α·Adam(ω,d ω ), where Adam is the gradient descent algorithm, α is the learning rate, and d ω is the gradient update of the network parameter ω.

6. The product defect detection method based on parallel multi-scale fusion network according to claim 1 is characterized in that: The expression of the fusion detection module is: Where σ is the sigmoid activation function, and Y is the defect area predicted by the product defect detection network model.

Images