Intelligent defect detection method and intelligent defect detection network based on S-shaped dynamic convolution and cross-weighted feature pyramid

Abstract

The invention provides an intelligent defect detection method and an intelligent defect detection network based on S-shaped dynamic convolution and a cross-weighted feature pyramid, and the method comprises the following steps: collecting an OK sample image and a plurality of NG sample images, and employing a co-occurrence matrix matching method based on Laplacian gradient features, extracting defect labeling area images of the plurality of NG sample images; constructing a feature extraction network based on S-type dynamic convolution for extracting defect features; constructing a pyramid module based on cross-weighted features, wherein the pyramid module is used for multi-level fusion calculation; building an intelligent defect detection network based on S-type dynamic convolution and a cross-weighted feature pyramid, and training the intelligent defect detection network; performing product detection by using the defect intelligent detection network based on the S-shaped dynamic convolution and the cross-weighted feature pyramid; through the arrangement, high-precision segmentation can be better carried out on the defect area and the background area, the precision and accuracy of industrial AOI defect detection are improved, and the quality safety of industrial AOI products is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to an intelligent defect detection method and an intelligent defect detection network based on S-type dynamic convolution and cross-weighted feature pyramid. Background Art

[0002] In the field of industrial AOI defect detection, due to the objective imaging differences between different batches of products and the randomness and probability of defect morphology, the 2D visual inspection technology cannot achieve good compatibility and applicability of product defect detection. In addition, due to the warping or uneven material of the product itself, the complexity of adjusting the 2D visual inspection parameters cannot meet the timeliness of industrial AOI defect detection. The intelligent defect detection method based on deep learning neural network can better extract high-level semantic information and low-level position information in product images. Through forward calculation and back propagation, it can better realize the task of segmenting and extracting defect area characteristic information. However, in the existing intelligent defect detection method, the convolution kernel image characteristic information extraction based on the sliding window cannot pay more attention to the defect area information, and cannot effectively capture the global information, resulting in the identification of defect areas is not comprehensive enough. In addition, the existing technology lacks intelligent detection methods for narrow and irregular defects such as cracks / scratches.

[0003] Therefore, it is necessary to design a defect intelligent detection method and defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid. Taking the occurrence morphology of crack / scratch defects as a reference, the calculation of the S-type dynamic convolution parameter matrix BestMatrix_S is completed in advance before the crack / scratch defect detection and recognition network starts training. Finally, a defect detection and recognition network based on BestMatrix_S and cross-weighted feature pyramid is built, and the defect segmentation area is acquired through parameter setting. A 2D image defect quality grade evaluation module is introduced to complete the OK / NG category judgment of the product to be inspected.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid, which comprises the following steps:

[0006] S1. Collect one OK sample image and multiple NG sample images, and use the co-occurrence matrix matching method based on Laplace gradient features to extract defect annotation area images of multiple NG sample images;

[0007] S2, construct a feature extraction network based on S-type dynamic convolution to extract defect features;

[0008] S3, build a cross-weighted feature pyramid module for multi-level fusion calculation;

[0009] S4. Build and train a defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid;

[0010] S5. Use the defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid to perform product detection.

[0011] As a further improved technical solution of the present invention, step S1 includes:

[0012] S11, collecting one OK sample image and obtaining an OK product area image based on the product edge contour;

[0013] S12, collecting multiple NG sample images and using them to obtain an NG product area image based on the product edge contour;

[0014] S13, obtaining Laplace gradient feature information of the OK product area, cropping the NG sample image and the NG product area image according to the product area information, and obtaining the NG sample image and the defect marking area image with background interference removed.

[0015] As a further improved technical solution of the present invention, in step S1, the co-occurrence matrix matching method based on Laplace gradient features includes:

[0016] The Laplace operation is performed on the OK product area image, which is described by formula (1):

[0017] L(x,y)=Σ[f(x',y')-f(x,y)] (1)

[0018] Where (x, y) is the horizontal and vertical coordinate value of the pixel point in the OK product area image, L(x, y) is the Laplace gradient eigenvalue at the pixel point (x, y) in the OK product area image, and f(x', y') is the grayscale value of the surrounding pixel point f(x, y);

[0019] After obtaining the Laplace gradient feature information of the OK product area image, the corresponding width and height scale information wh is calculated, and the search window size is set to w+Δw,h+Δh in multiple NG sample images, where Δw and Δh are decimal constants in the range of 0-1, which are used to adjust the relative position of the search window in the search image. The Laplace gradient feature information in the search window area is calculated, and the similarity is calculated with the gradient feature information of the OK product area image. The search window position with the highest similarity is selected as the product ROI area, marked as (leftx, lefty, rightx, righty), where (leftx, lefty) is the horizontal and vertical coordinates of the upper left pixel of the product ROI area in the image, and (rightx, righty) is the horizontal and vertical coordinates of the upper right pixel of the product ROI area in the image. The product ROI area coordinates are used to crop the NG sample image and the NG product area image respectively, and the NG sample image and the defect annotation area image with background interference removed are further obtained.

[0020] As a further improved technical solution of the present invention, step S2 includes:

[0021] S21, using five 3×3 convolution kernels and one 7×7 convolution kernel as the basic module of S-type dynamic convolution, taking the NG sample image after removing background interference as input, the defect annotation area as output, and synchronously extracting the bone line of the defect annotation area as the moving trajectory of the center point of the S-type dynamic convolution kernel;

[0022] S22. Optimize the S-type dynamic convolution module parameters through the forward calculation and reverse feedback of the model to obtain the S-type dynamic convolution parameter matrix.

[0023] As a further improved technical solution of the present invention, in step S2, the bone line contour information of the defect area image is obtained by formula (3), and formula (3) is:

[0024] setp1ascenter,p2-p6asadjacentpixel;

[0025] if N(p1)∈[2,6] and S(p1)==1 then p2p4p6=0 and p4p6p8=0 (3)

[0026] In formula (2), p1 is a pixel in the defect area image, p2, p4, p6, and p8 correspond to the adjacent pixels in the upper, right, lower, and left directions of p1, respectively. N(p1) is the number of foreground pixels among the eight pixels adjacent to p1, and S(p1) is the cumulative number of times 0-1 appears in the eight adjacent pixels, where 0 is the background and 1 is the foreground, i.e., the defect area. After looping through the defect area image, the position where 1 occurs is the bone line contour information of the current defect area.

[0027] After the image information is initially extracted by five 3×3 small convolutional layers, the starting point of the bone line contour area in the defect area image, i.e., the point with the smallest horizontal and vertical pixel coordinate values, is used as the initial point for calculating the 7×7 convolution kernel. The offsets of the 7×7 convolution kernel in the horizontal and vertical directions of the defect area image are simultaneously set to Δx and Δy, where the initial values ​​of Δx and Δy are decimal constants between 0 and 1. The Euclidean distance between the next offset point of the convolution kernel and the bone line area is calculated, and the values ​​of Δx and Δy are continuously adjusted. After the 7×7 convolution kernel traverses the entire defect area image through the bone line area contour, each step of the offset in the horizontal and vertical directions of the defect area image is recorded as the S-type dynamic convolution parameter matrix BestMatrix_S.

[0028] As a further improved technical solution of the present invention, step S3 includes:

[0029] S31, add 5 layers of feature pyramid, corresponding to P1, P2, P3, P4, P5 from bottom to top, and the upper layer and the lower layer are connected with a ×2 sampling relationship; add 5 layers of feature pyramid, corresponding to P10, P9, P8, P7, P6 from top to bottom, and the upper layer and the lower layer are connected with a ×1 / 2 sampling relationship; among them, P1 and P10, P2 and P9, P3 and P8, P4 and P7, P5 and P6 are connected with the set weight ratio coefficient, and the upper network can receive high-resolution semantic information from the lower layer through multi-scale and multi-feature fusion, and obtain high-quality semantic information in the process of feature fusion;

[0030] S32, the channel attention mechanism is introduced into the bottom-up and top-down pyramid cross-weighted construction connection module to select key channel information and modify the higher weight ratio to complete the defect prediction based on S-type dynamic convolution and cross-weighted feature pyramid;

[0031] The calculation method of the channel attention mechanism is described by formula (4):

[0032]

[0033] In formula (4), M_Channel(F) is the attention channel mechanism calculation matrix, σ is the adjustment coefficient, and They represent the global average pooling features and the maximum pooling features respectively, and ω0 and ω1 correspond to the weight coefficient parameters respectively.

[0034] As a further improved technical solution of the present invention, step S4 includes the following steps:

[0035] S41. Build a defect training and recognition neural network based on S-type dynamic convolution and cross-weighted feature pyramid;

[0036] S42, determining the hyperparameters of the model, wherein the hyperparameters include a loss function, a training iteration cycle, and a learning rate;

[0037] S43, perform model training, solidify and export network parameters after training, and obtain a model file.

[0038] As a further improved technical solution of the present invention, in step S4, the model is set to calculate the regression loss function through formula (5), and the calculation method is:

[0039]

[0040] Among them, p(x, y) and q(x, y) correspond to the original annotation true value information and the predicted information calculated based on the existing network, respectively, and γ is a hyperparameter for adjusting the weight coefficient;

[0041] By setting the number of iteration cycles of the neural network model calculation, calculating the loss threshold and learning rate, the parameter matrix of the last calculation of each layer is solidified and saved after completing the model training, and the model file Model is exported, which contains the S-type dynamic convolution parameter matrix.

[0042] As a further improved technical solution of the present invention, step S5 includes the following steps:

[0043] S51, the newly collected product image is recorded as Image_i (i=1, 2, 3, ..., n) as the model calculation input end;

[0044] S52, the model outputs defect area information of the product image;

[0045] S53, designing a defect grade quality assessment module based on 2D vision, which is used to assess defect region information, wherein the defect region information includes defect area and defect length, and the defect area and defect length are calculated using formula (6).

[0046]

[0047] Among them, x, y correspond to (x, y) pixels at different coordinate positions, adjacent pixels have different grayscale values, and a series of two-dimensional coordinate points are statistically obtained, recorded as the defect area contour Contour, and the area corresponding to the Contour and the longest side of the minimum circumscribed rectangle, i.e., the length information, are obtained;

[0048] S54. Determine the OK / NG category of the product based on the evaluation results.

[0049] The object of the present invention is to provide a defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid to complete crack / scratch defect detection of a product to be inspected.

[0050] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid as described in any of the above items, which includes a feature extraction network, a feature fusion network and a feature prediction network, wherein the feature extraction network includes 5 3×3 convolution layers and 1 pooling layer, 1 S-type dynamic convolution module and 1 fully connected layer, wherein the 5 3×3 convolution layers are used to preliminarily extract image feature information, 1 pooling layer is used for feature information dimensionality reduction, 1 S-type dynamic convolution module is used to calculate the convolution feature calculation matrix according to preset bone line contour information, and 1 fully connected layer is used for feature information dimension conversion to retain all useful information, the feature fusion network includes a bottom-up feature pyramid and a top-down feature pyramid, the feature prediction network includes 4 deconvolution layers, 1 Sigmoid layer and 1 Threshold extraction layer, wherein the 4 deconvolution layers are used for upsampling operations of feature information maps predicted by feature pyramids of different scales, and 1 Sigmoid layer is used to extract the category and position information of the defect prediction box

[0051] It can be seen from the above technical solutions that the present invention takes the most common defects in the AOI field - cracks / scratches as the research object. Considering that the morphology of crack defects is mostly "S" shaped, the neural network intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid of the present invention replaces the existing fixed kernel size convolution with S-type dynamic convolution, and uses the bone line of the defect area as the moving trajectory of the convolution kernel to better extract the feature information that conforms to the crack / scratch defect. At the same time, in order to deeply fuse the low-level and high-level defect feature map information, a cross-weighted feature pyramid module is proposed to perform fusion calculations on feature map information of different levels with different weight ratios, so as to better perform high-precision segmentation of defect areas and background areas, improve the precision and accuracy of industrial AOI defect detection, and ensure the quality and safety of industrial AOI products. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1Shown is the architecture diagram of the defect intelligent detection network of the S-shaped dynamic convolution and cross-weighted feature pyramid of the present invention;

[0053] Figure 2 Shown is an image of the product area obtained during actual inspection;

[0054] Figure 3 The image shown is the defect marking area image obtained in the actual inspection;

[0055] Figure 4 Shown is a schematic diagram of defect level assessment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the "S-type" described in the present invention does not represent a strict limitation on the defect shape, that is, the defect shape applicable to the present invention is not limited to the "S-type", but also includes other shapes such as curves, arc lines, broken lines, etc. similar to the "S" shape. This article only uses "S-type" to refer to the above shapes.

[0057] The present invention provides a defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid, and the specific steps are as follows:

[0058] S1. Collect one OK sample image and multiple NG sample images, and use the co-occurrence matrix matching method based on Laplace gradient features to extract defect annotation area images of multiple NG sample images;

[0059] S2, construct a feature extraction network based on S-type dynamic convolution to extract defect features;

[0060] S3, build a cross-weighted feature pyramid module for multi-level fusion calculation;

[0061] S4. Build and train a defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid;

[0062] S5. Use the defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid to perform product detection.

[0063] Specifically, step S1 includes:

[0064] S11, manually collect an OK sample image and obtain an OK product area image based on the product edge contour, so as to mark and obtain the contour area divided by the product and the background;

[0065] S12, collect 10 NG sample images and mark the corresponding crack / scratch defect areas: and use them to obtain NG product area images based on product edge contours; in S11 and S12, MVTec Deep Learning Tool 23.08EarlyAdopter (version number: 23.08) software is used to complete the delineation of the product and background boundaries in the sample images;

[0066] S13. Take the image grayscale co-occurrence matrix based on Laplace gradient feature information to extract 10 NG area images and defect marking area images: obtain the Laplace gradient feature information of the OK product area, crop the NG sample image and NG product area image according to the product area information, and obtain the NG sample image and defect marking area image with background interference removed.

[0067] In step S13, first, the Laplace gradient feature information of the current OK product area image is calculated. The calculation method can be described by formula (1):

[0068] L(x,y)=Σ[f(x',y')-f(x,y)] (1)

[0069] Wherein, (x, y) is the horizontal and vertical coordinate value of the pixel point in the OK product area image, L(x, y) is the Laplace gradient eigenvalue at the pixel point (x, y) in the OK product area image, and f(x', y') is the grayscale value of the surrounding pixel point f(x, y). Further, after obtaining the Laplace gradient feature information of the OK product area image, the corresponding width and height scale information wh is calculated, and the search window size is set to w+Δw,h+Δh in the 10 NG sample images, where Δw and Δh are decimal constants in the range of 0-1, which are used to adjust the relative position of the search window in the search image. By calculating the Laplace gradient feature information L'(x, y) in the search window area, the similarity calculation is performed with the gradient feature information of the OK product area image, that is, comparing the dimension difference distance of the characteristic information L(x, y) and L'(x, y). The calculation method can be described by formula (2).

[0070]

[0071] Among them, Dis is used to describe the Laplace gradient feature and grayscale co-occurrence matrix information dimension distance between the standard OK product ROI area and the NG product sliding window area, and the two feature information account for 85% and 15% of the weight ratio coefficients respectively. The present invention introduces the grayscale co-occurrence matrix to calculate the grayscale value difference within the search window field of view. Since there is a certain degree of difference in the spatial posture of the product incoming material placement position, a smaller calculation weight system is set in the process of calculating the feature information difference, and the Laplace gradient feature information is still used as the main feature information distance calculation reference; the optimal search window of the NG product image is completed by formula (2), that is, the product area positioning in the NG sample image is completed.

[0072] Furthermore, the search window position with the highest similarity is selected as the product ROI area, marked as (leftx, lefty, rightx, righty), where (leftx, lefty) is the horizontal and vertical coordinates of the upper left pixel of the product ROI area in the image, and (rightx, righty) is the horizontal and vertical coordinates of the upper right pixel of the product ROI area in the image. The ROI area is removed synchronously in the NG sample image NGImage_i (i=1,2,3,...,10) and the NG product area image NGMaskScratch_i (i=1,2,3,...,10), and the NG sample image and defect annotation area image with background interference removed are obtained.

[0073] Step S2 includes:

[0074] S21, using five 3×3 convolution kernels and one 7×7 convolution kernel as the basic module of S-type dynamic convolution, taking the NG sample image after removing background interference as input, the defect annotation area as output, and synchronously extracting the bone line of the defect annotation area as the moving trajectory of the center point of the S-type dynamic convolution kernel;

[0075] In step S2, the bone line contour information of the defect area image is obtained by formula (3), and formula (3) is:

[0076] setp1ascenter,p2-p6asadjacentpixel;

[0077] if N(p1)∈[2,6] and S(p1)==1 then p2p4p6=0 and p4p6p8=0 (3)

[0078] In formula (3), p1 is a pixel in the defect area image, p2, p4, p6, and p8 correspond to the adjacent pixels in the upper, right, lower, and left directions of p1, respectively. N(p1) is the number of foreground pixels among the eight pixels adjacent to p1, and S(p1) is the cumulative number of times 0-1 appears in the eight adjacent pixels, where 0 is the background and 1 is the foreground, i.e., the defect area. After looping through the defect area image, the position where 1 occurs is the bone line contour information of the current defect area.

[0079] S22. After the image information is initially extracted by five 3×3 small convolutional layers, the starting point of the bone line contour area in the defect area image, i.e., the point with the smallest horizontal and vertical pixel coordinate values, is used as the initial point for calculating the 7×7 convolution kernel. The horizontal and vertical offsets of the 7×7 convolution kernel in the defect area image are simultaneously set to Δx and Δy, where the initial values ​​of Δx and Δy are decimal constants between 0 and 1. The Euclidean distance between the next offset point of the convolution kernel and the bone line area is calculated, and the S-type dynamic convolution moving position corresponding to the nearest distance is selected. The values ​​of Δx and Δy are continuously adjusted. After the 7×7 convolution kernel traverses the entire defect area image through the bone line area contour, each step of the offset in the horizontal and vertical directions of the defect area image is recorded as the S-type dynamic convolution parameter matrix BestMatrix_S. Compared with the fixed kernel size convolution operation, the S-type dynamic convolution pays more attention to the central area of ​​the crack / scratch defect by calculating the offset at different positions in the horizontal and vertical directions of the image. The crack / scratch defect is located at the center of the 7×7 convolution kernel. In this step, based on the image information of 10 confirmed crack defect areas, the S-type dynamic convolution module parameters are optimized through the forward calculation and reverse feedback of the model to obtain the S-type dynamic convolution parameter matrix.

[0080] Step S3 includes:

[0081] S31. After completing the calculation of the best S-type dynamic convolution matrix BestMatrix_S, add 5 layers of feature pyramids, corresponding to P1, P2, P3, P4, and P5 from bottom to top, and the upper layer and the lower layer are connected with a ×2 sampling relationship; further add 5 layers of feature pyramids, corresponding to P10, P9, P8, P7, and P6 from top to bottom, and the upper layer and the lower layer are connected with a ×1 / 2 sampling relationship; P1 and P10, P2 and P9, P3 and P8, P4 and P7, P5 and P6 are connected with the set weight ratio coefficient, and the upper network can receive high-resolution semantic information from the lower layer through multi-scale and multi-feature fusion, and obtain high-quality semantic information in the process of feature fusion;

[0082] S32, the channel attention mechanism is introduced into the bottom-up and top-down pyramid cross-weighted construction connection module to select key channel information and modify the higher weight ratio to complete the defect prediction based on S-type dynamic convolution and cross-weighted feature pyramid;

[0083] The calculation method of the channel attention mechanism can be described by formula (4):

[0084]

[0085] In formula (4), M_Channel(F) is the attention channel mechanism calculation matrix, σ is the adjustment coefficient, and They represent the global average pooling features and the maximum pooling features respectively. ω0 and ω1 correspond to the weight coefficient parameters respectively. By calculating the construction of top-down and bottom-up feature pyramids, and the connection between different layers based on certain weight coefficients, the channel attention mechanism can effectively extract important channel information, set higher weight ratio parameters, and improve the accuracy of multi-scale feature information fusion.

[0086] Step S4 includes the following steps:

[0087] S41. Build a defect training and recognition neural network based on S-type dynamic convolution and cross-weighted feature pyramid, including the above-mentioned feature extraction network (5 3×3 convolution layers and 1 pooling layer, 1 S-type dynamic convolution module and 1 fully connected layer, where 5 3×3 convolution layers are used to preliminarily extract image feature information, 1 pooling layer is used for feature information dimensionality reduction, 1 S-type dynamic convolution module is used to calculate the convolution feature calculation matrix according to the preset bone line contour information, and 1 fully connected layer is used for feature information dimension conversion to retain all useful information), feature fusion network (a bottom-up feature pyramid and a top-down feature pyramid) The invention relates to a feature prediction network (4 deconvolution layers, 1 Sigmoid layer and 1 Threshold extraction layer, wherein the 4 deconvolution layers are used for upsampling operations of feature information maps predicted by feature pyramids of different scales, and finally feature dimension information consistent with the size of the original input image is obtained, and the 1 Sigmoid layer is used to extract the category and position information of the defect prediction box. Since there is only one defect category information, i.e., cracks / scratches, in the present invention, the last Threshold extraction layer is used to set appropriate threshold extraction parameters to accurately segment the defect contour information);

[0088] S42, determining the hyperparameters of the model, wherein the hyperparameters include a loss function, a training iteration cycle, and a learning rate; wherein the model can be set to calculate the regression loss function by formula (5), and the calculation method is:

[0089]

[0090] Among them, p(x, y) and q(x, y) correspond to the original annotation true value information and the predicted information calculated based on the existing network, respectively. γ is a hyperparameter for adjusting the weight coefficient, which includes three types of loss functions, corresponding to the cross entropy loss calculation (setting the weight coefficient to 0.3), L1 distance loss calculation (setting the weight coefficient to 0.2) and focal loss function calculation (setting the weight coefficient to 0.5). The three loss function calculations are comprehensively considered and different weight coefficients are set to calculate the parameter update in the back propagation process of the model.

[0091] S43. By setting the number of iteration cycles of the neural network model calculation, calculating the loss threshold and the learning rate, after completing the model training, the parameter matrix of the last calculation of each layer is solidified and saved, and the model file Model is exported.

[0092] Step S5 includes:

[0093] S51, the newly collected product image is recorded as Image_i (i=1,2,3,...,n) as the model calculation input terminal, please refer to Figure 2 As shown;

[0094] S52, set the threshold of the last layer of Threshold extraction layer to 128, that is, the pixels with grayscale values ​​> 128 are regarded as crack / scratch defect areas, and the pixels with grayscale values ​​≤ 128 are regarded as background areas. The model outputs the defect area information of the product image (see Figure 3 As shown), the defect area information includes the defect area DefectArea and the defect length DefectLength;

[0095] S53, designing a defect level quality assessment module based on 2D vision, which is used to assess defect area information: after obtaining the defect area contour information, setting the defect area DefectArea and the defect length DefectLength as reference indicators of the defect level quality assessment module;

[0096] It can be calculated using formula (6):

[0097]

[0098] Formula (6) is used to calculate the grayscale values ​​of adjacent pixels at different (x, y) pixel coordinate positions in the image, and statistically obtain a series of two-dimensional coordinate points, which are recorded as the defect area contour Contour. Further, the area of ​​the corresponding Contour and the longest side of the minimum circumscribed rectangle, i.e., the length information, can be obtained;

[0099] S54, set different product OK / NG judgment labels to obtain the OK / NG product category of the current product: the NG product judgment standard in the actual product online detection is: DefectArea>4mm^2, DefectLength>1mm, and statistically analyze the quantitative index information of different product defect areas to effectively judge the OK / NG category. Figure 4 As shown, after evaluation, the defect area of ​​the defective product is 70mm^2 (>4mm^2) and the length is 13mm (>1mm), so the output evaluation result is NG.

[0100] In summary, the present invention takes the most common defects in the field of industrial AOI detection, namely cracks / scratches, as the research object, and takes the defect morphology of cracks / scratches as a reference, proposes an image defect feature extraction network based on S-type dynamic convolution, takes the defect area bone line as the reference of the convolution kernel moving trajectory, sets the offsets Δx and Δy in different directions, and adjusts the values ​​of Δx and Δy in real time during the calculation process, adjusts the convolution kernel moving trajectory, and finally obtains the parameter calculation matrix BestMatrix_S based on S-type convolution. At the same time, a cross-weighted feature pyramid is proposed for weighted fusion of image feature information of different layers, so that the upper network can receive higher-precision semantic information from the lower layer, adds an attention channel module, and adjusts the weight coefficient in real time during the back propagation process of the model. A deep learning detection neural network model for crack / scratch defects based on S-type dynamic convolution and cross-weighted feature pyramid is built, and an intelligent detection method for crack / scratch defects is proposed. By introducing a 2D image defect grade evaluation module, accurate judgment of the OK / NG category of different batches of products is completed.

[0101] The above embodiments are only used to illustrate the present invention but not to limit the technical solutions described in the present invention. The understanding of this specification should be based on the technicians in the relevant technical field. Although this specification has described the present invention in detail with reference to the above embodiments, ordinary technicians in the field should understand that technicians in the relevant technical field can still modify or replace the present invention with equivalents, and all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid, characterized in that: The following steps are involved: S1. Collect one OK sample image and multiple NG sample images, and use the co-occurrence matrix matching method based on Laplace gradient features to extract defect annotation area images of multiple NG sample images; S2, construct a feature extraction network based on S-type dynamic convolution to extract defect features; S3, build a cross-weighted feature pyramid module for multi-level fusion calculation; S4. Build and train a defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid; S5. Use the defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid to perform product detection.

2. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 1, characterized in that: Step S1 includes: S11, collecting one OK sample image and obtaining an OK product area image based on the product edge contour; S12, collecting multiple NG sample images and using them to obtain an NG product area image based on the product edge contour; S13, obtaining Laplace gradient feature information of the OK product area, cropping the NG sample image and the NG product area image according to the product area information, and obtaining the NG sample image and the defect marking area image with background interference removed.

3. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid as claimed in claim 2, characterized in that: In step S1, the co-occurrence matrix matching method based on Laplace gradient features includes: The Laplace operation is performed on the OK product area image, which is described by formula (1): L(x,y)=Σ[f(x',y')-f(x,y)] (1) Where (x, y) is the horizontal and vertical coordinate value of the pixel point in the OK product area image, L(x, y) is the Laplace gradient eigenvalue at the pixel point (x, y) in the OK product area image, and f(x', y') is the grayscale value of the surrounding pixel point f(x, y); After obtaining the Laplace gradient feature information of the OK product area image, the corresponding width and height scale information wh is calculated, and the search window size is set to w+Δw,h+Δh in multiple NG sample images, where Δw and Δh are decimal constants in the range of 0-1, which are used to adjust the relative position of the search window in the search image. The Laplace gradient feature information in the search window area is calculated, and the similarity is calculated with the gradient feature information of the OK product area image. The search window position with the highest similarity is selected as the product ROI area, marked as (leftx, lefty, rightx, righty), where (leftx, lefty) is the horizontal and vertical coordinates of the upper left pixel of the product ROI area in the image, and (rightx, righty) is the horizontal and vertical coordinates of the upper right pixel of the product ROI area in the image. The product ROI area coordinates are used to crop the NG sample image and the NG product area image respectively, and the NG sample image and the defect annotation area image with background interference removed are further obtained.

4. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid as claimed in claim 3, characterized in that: Step S2 includes: S21, using five 3×3 convolution kernels and one 7×7 convolution kernel as the basic module of S-type dynamic convolution, taking the NG sample image after removing background interference as input, the defect annotation area as output, and synchronously extracting the bone line of the defect annotation area as the moving trajectory of the center point of the S-type dynamic convolution kernel; S22. Optimize the S-type dynamic convolution module parameters through the forward calculation and reverse feedback of the model to obtain the S-type dynamic convolution parameter matrix.

5. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 4, characterized in that: In step S2, the bone line contour information of the defect area image is obtained by formula (3), and formula (3) is: setp1ascenter,p2-p6asadjacentpixel; if N(p1)∈[2,6] and S(p1)==1 then p2p4p6=0 and p4p6p8=0 (3) In formula (2), p1 is a pixel in the defect area image, p2, p4, p6, and p8 correspond to the adjacent pixels in the upper, right, lower, and left directions of p1, respectively. N(p1) is the number of foreground pixels among the eight pixels adjacent to p1, and S(p1) is the cumulative number of times 0-1 appears in the eight adjacent pixels, where 0 is the background and 1 is the foreground, i.e., the defect area. After looping through the defect area image, the position where 1 occurs is the bone line contour information of the current defect area. After the image information is initially extracted by five 3×3 small convolutional layers, the starting point of the bone line contour area in the defect area image, i.e., the point with the smallest horizontal and vertical pixel coordinate values, is used as the initial point for calculating the 7×7 convolution kernel. The offsets of the 7×7 convolution kernel in the horizontal and vertical directions of the defect area image are simultaneously set to Δx and Δy, where the initial values ​​of Δx and Δy are decimal constants between 0 and 1. The Euclidean distance between the next offset point of the convolution kernel and the bone line area is calculated, and the values ​​of Δx and Δy are continuously adjusted. After the 7×7 convolution kernel traverses the entire defect area image through the bone line area contour, each step of the offset in the horizontal and vertical directions of the defect area image is recorded as the S-type dynamic convolution parameter matrix BestMatrix_S.

6. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 5, characterized in that: Step S3 includes: S31, add 5 layers of feature pyramid, corresponding to P1, P2, P3, P4, P5 from bottom to top, and the upper layer and the lower layer are connected with a ×2 sampling relationship; add 5 layers of feature pyramid, corresponding to P10, P9, P8, P7, P6 from top to bottom, and the upper layer and the lower layer are connected with a ×1 / 2 sampling relationship; among them, P1 and P10, P2 and P9, P3 and P8, P4 and P7, P5 and P6 are connected with the set weight ratio coefficient, and the upper network can receive high-resolution semantic information from the lower layer through multi-scale and multi-feature fusion, and obtain high-quality semantic information in the process of feature fusion; S32, the channel attention mechanism is introduced into the bottom-up and top-down pyramid cross-weighted construction connection module to select key channel information and modify the higher weight ratio to complete the defect prediction based on S-type dynamic convolution and cross-weighted feature pyramid; The calculation method of the channel attention mechanism is described by formula (4): In formula (4), M_Channel(F) is the attention channel mechanism calculation matrix, σ is the adjustment coefficient, and They represent the global average pooling features and the maximum pooling features respectively, and ω0 and ω1 correspond to the weight coefficient parameters respectively.

7. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 1, characterized in that: Step S4 includes the following steps: S41. Build a defect training and recognition neural network based on S-type dynamic convolution and cross-weighted feature pyramid; S42, determining the hyperparameters of the model, wherein the hyperparameters include a loss function, a training iteration cycle, and a learning rate; S43, perform model training, solidify and export network parameters after training, and obtain a model file.

8. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 7, characterized in that: In step S4, the model is set to calculate the regression loss function through formula (5), and the calculation method is: Among them, p(x, y) and q(x, y) correspond to the original annotation true value information and the predicted information calculated based on the existing network, respectively, and γ is a hyperparameter for adjusting the weight coefficient; By setting the number of iteration cycles of the neural network model calculation, calculating the loss threshold and learning rate, the parameter matrix of the last calculation of each layer is solidified and saved after completing the model training, and the model file Model is exported, which contains the S-type dynamic convolution parameter matrix.

9. The defect intelligent detection method based on S-type dynamic convolution and cross-weighted feature pyramid according to claim 1, characterized in that: Step S5 includes the following steps: S51, the newly collected product image is recorded as Image_i (i=1, 2, 3, ..., n) as the model calculation input end; S52, the model outputs defect area information of the product image; S53, designing a defect grade quality assessment module based on 2D vision, which is used to assess defect region information, wherein the defect region information includes defect area and defect length, and the defect area and defect length are calculated using formula (6). Among them, x, y correspond to (x, y) pixels at different coordinate positions, adjacent pixels have different grayscale values, and a series of two-dimensional coordinate points are statistically obtained, recorded as the defect area contour Contour, and the area corresponding to the Contour and the longest side of the minimum circumscribed rectangle, i.e., the length information, are obtained; S54. Determine the OK / NG category of the product based on the evaluation results.

10. A defect intelligent detection network based on S-type dynamic convolution and cross-weighted feature pyramid, characterized in that: The intelligent defect detection network includes a feature extraction network, a feature fusion network and a feature prediction network. The feature extraction network includes 5 3×3 convolutional layers and 1 pooling layer, 1 S-type dynamic convolution module and 1 fully connected layer, wherein the 5 3×3 convolutional layers are used to preliminarily extract image feature information, 1 pooling layer is used for feature information dimensionality reduction, 1 S-type dynamic convolution module is used to calculate the convolution feature calculation matrix according to preset bone line contour information, and 1 fully connected layer is used for feature information dimension conversion to retain all useful information. The feature fusion network includes a bottom-up feature pyramid and a top-down feature pyramid. The feature prediction network includes 4 deconvolution layers, 1 Sigmoid layer and 1 Threshold extraction layer, wherein the 4 deconvolution layers are used for upsampling operations of feature information maps predicted by feature pyramids of different scales, and 1 Sigmoid layer is used to extract the category and position information of the defect prediction box.

Images