Component weak defect high-precision detection method based on faster-rcnn model

Abstract

The application discloses a kind of weak defect high-precision detection methods of component based on Faster-RCNN model, comprising:1) component image acquisition;2) bilinear interpolation is carried out to original image, and the superscale graph that is twice size of original graph is obtained;3) feature extraction is carried out to original graph, and the feature of original graph is obtained;Super scale graph is extracted, and the feature of super scale graph is obtained;4) maximum value downsampling is carried out to super scale graph feature, and the feature of original graph is aligned;5) super scale graph feature and the feature of original graph are spliced and fused;6) the feature map after fusion is input detection head, and the positioning and classification of defect are carried out.The application detects electronic component defect based on Faster-RCNN model, strengthens the detection ability to weak defect, alleviates the problem that prior art weak defect detection precision of component is not high, improves the overall detection precision of component defect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial electronic component defect detection, in particular to a component weak defect high-precision detection method based on a Faster-RCNN model. BACKGROUND

[0002] Manual screening is the earliest detection method for industrial products, but this method has low accuracy, poor real-time performance, low efficiency, high cost, and is greatly affected by human experience and subjective factors. The automatic detection method based on machine vision can overcome the above-mentioned drawbacks to a great extent and performs outstandingly in the automatic production of electronic component products. Through the cooperation of tray, turntable, high-speed camera, light source and other equipment, a hardware mechanism for electronic component defect detection can be formed. Timely detection of appearance defects can effectively improve the overall quality of electronic components, prolong the service life and enhance the competitiveness of enterprises.

[0003] At present, the detection method based on machine vision can be divided into image processing method, traditional machine learning method and deep learning method. The image processing method usually needs manual feature design, is highly dependent on experience, and has poor generalization ability for different tasks, so it has gradually withdrawn from the research field of people. With the continuous improvement of the performance requirements of industrial vision detection, the detection method based on traditional machine learning has been difficult to meet the requirements of modern industry in high precision and speed. The deep learning method is superior to the traditional machine learning method in detection precision and speed. However, the general deep learning method backbone has a large downsampling rate, which is easy to lose detailed information and is not conducive to weak defect detection. The existing method detects weak defects through deep and shallow feature map fusion and feature pyramid network. However, very weak defects may only have a few dozen pixels, and after extraction by the shallow network, many useful detailed information is still lost, which cannot cope with the detection of very weak defects.

[0004] Therefore, the current detection methods are difficult to cope with the weak defects widely existing in industrial products, and cannot meet the ultra-high precision detection requirements of modern industry. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a component weak defect high-precision detection method based on a Faster-RCNN model, to realize high-precision automatic detection of industrial products, improve the overall quality of industrial electronic components, and enhance the competitiveness of enterprises.

[0006] To achieve the above purpose, the technical scheme provided by the present application is: a component weak defect high-precision detection method based on a Faster-RCNN model, comprising the following steps:

[0007] 1) Collecting the original image of the component to be detected;

[0008] 2) Bilinear interpolation is performed on the original image of step 1) to obtain a superscale image with a size twice that of the original image;

[0009] 3) The original image is taken as the input of the first stream of the Faster-RCNN model, and feature extraction is performed through the entire feature extraction module to obtain original image features containing high-level semantic information;

[0010] The superscale image is taken as the input of the second stream of the Faster-RCNN model, and feature extraction is performed through the first three layers of the feature extraction module to retain the detailed information in the superscale image that is conducive to weak defect detection, thereby obtaining superscale image features;

[0011] 4) Maximum downsampling is performed on the superscale image features to align the size with the original image features;

[0012] 5) The superscale image features obtained in the previous step are spliced with the original image features, and a 1*1 convolution kernel is used to perform channel transformation and fusion on the spliced feature map;

[0013] 6) The fused feature map in the previous step is input into the detection head of the Faster-RCNN model to locate and classify the component defects, and the defect detection information is output.

[0014] Further, in step 1), the component to be detected is placed in a tray, and after the tray is shaken, the component is orderly placed on a transparent disc. The disc sends the component to be detected to the camera. A hardware triggering method is used to control the light source and the camera to obtain the original image of the component to be detected.

[0015] Further, in step 2), bilinear interpolation is performed on the original image to enlarge the image size to twice the original size, thereby obtaining a superscale image to enlarge the area of weak defects and make it easier for the Faster-RCNN model to find weak defects;

[0016] The process of bilinear interpolation is as follows:

[0017] f(x,y)=w(Q 11 )(x2-x)(y2-y)+w(Q 21 )(x-x1)(y2-y)+w(Q 12 )(x2-x)(y-y1)+w(Q 22 )(x-x1)(y-y1)

[0018]

[0019] In the formula, f(x, y) is the pixel value of the interpolated hyperscale map at the coordinates (x, y), x is the horizontal coordinate, y is the vertical coordinate, x1, x2, y1, y2 are the horizontal and vertical coordinates of the nearest neighbor points, Q ij ij i j ij 11 12 21 22

[0020] Further, in step 3), Faster-RCNN is used as a basic model, Resnet50 is used as a feature extraction module to extract the features of the original image and the hyperscale map respectively; Resnet50 is used to extract the features of the original image to obtain the original image features containing rich semantic information; in order to retain the detailed information in the hyperscale map that is beneficial to defect detection, only the first three layers of Resnet50 are used to extract the features of the hyperscale map to obtain the hyperscale map features containing shallow detailed information; Resnet50 is composed of convolutional layers, pooling layers and nonlinear mapping;

[0021] The convolution process is as follows:

[0022]

[0023] In the formula, f1[x, y] is the data of the image in the (x, y) region, w[x, y] is the convolution kernel, f2[x, y] is the obtained feature after convolution, n i j represent the offset distance from the convolution center, n1 and n2 are the maximum offset distances in the vertical and horizontal directions of convolution respectively, f[x+n i , y+n j ] is the value of the image in (x+n i , y+n j ), w[n i , n j ] is the weight of the convolution kernel in the (n i , n j ) position;

[0024] The nonlinear mapping process is as follows:

[0025] f3[x, y] = max(0, f2[x, y])​​​​​​​​​​

[0026] In the formula, f3[x, y] is a feature map obtained after a nonlinear mapping.

[0027] Further, in step 4), since the hyper-scale map feature scale obtained in the previous step is large, it needs to be scaled to the same size as the original map feature, so maximum pooling is used to retain important information in the feature map and remove redundant information.

[0028] Further, in step 5), the original map feature and the scaled hyper-scale map feature are spliced by channel, and then a 1*1 convolution kernel is used to transform the channel number on the basis of keeping the feature map size unchanged, so as to promote the information fusion between channels.

[0029] In the formula, the size of the feature map after convolution is calculated as follows:

[0030]

[0031] In the formula, W out is the size of the output feature map, W in is the size of the input feature map, padding is the number of padding, F represents the size of the convolution kernel, and stride is the step length of convolution; when padding is 0, stride = 1, and the size of the convolution kernel is 1, the size of the output feature map is unchanged, and the number of channels of the output feature map is determined by the number of 1*1 convolution kernels.

[0032] Further, in step 6), the feature map fused in the previous step is input into the detection head of the Faster-RCNN model, and the task of the detection head includes two tasks of regression and classification, and is divided into two stages; the first stage is the binary classification of foreground and background and the regression of the foreground frame, that is, to judge whether the anchor frame contains defects, and to regress the anchor frame containing defects to make the position more accurate; the second stage is the classification of the foreground frame and the second regression of the foreground frame, which classifies the types of defects in the frame, and the regression task is to make the position of the foreground frame more accurate to better cover the defects.

[0033] In the formula, the classification task uses a cross-entropy loss function, which has the following form:

[0034]

[0035] In the formula, L cls is the classification loss, N is the total number of samples, M is the total number of classes, y ic is a symbol function, c represents a class, and i represents a sample; if the true class of sample i is c, then y ic = 1, otherwise 0, P ic represents the predicted probability that sample i belongs to class c.

[0036] The regression task uses a smooth L1 loss as follows:

[0037]

[0038] In the formula, smooth L1 represents the total loss, and l represents the loss value of each sample.

[0039] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0040] 1. Compared with the traditional machine learning algorithm, the present application has higher precision, faster speed and stronger anti-interference ability.

[0041] 2. Compared with other deep learning methods, the present application increases the second stream based on the super-scale image. The super-scale image can increase the area of weak defects, but the disadvantage is that it may destroy the image detail information. Therefore, the present application uses the original image as the first stream input and the super-scale image as the second stream input; thereby using the super-scale image to assist the model to find weak defects, effectively enhancing the detection ability of the model to weak defects and improving the recall rate of weak defects, which is of great significance to improve the overall quality of electronic components.

[0042] 3. The present application adopts a skip connection feature fusion method, which can effectively retain the detail information in the shallow feature map that is beneficial to weak defects. The spliced image not only contains high-level semantic information, but also contains detail information, thereby widening the optimization path of the model and mining features with distinguishing degrees, thereby enhancing the robustness of the model.

[0043] 4. The present application has wide application space in the field of industrial machines and can realize end-to-end training and detection, has low requirements for the experience and professional knowledge of users, and has wide application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the total framework diagram of the present embodiment.

[0045] Figure 2 is the original image of the inductor of the present embodiment.

[0046] Figure 3 is the super-scale image of the inductor of the present embodiment.

[0047] Figure 4 is the detection result schematic diagram of the present embodiment. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0049] For example, Figures 1 to 4As shown, the embodiment discloses a component weak defect high-precision detection method based on a Faster-RCNN model, including the following steps:

[0050] 1) Place the 0402 patch inductor on the tray, start the switch, and the tray will shake off the inductor to the turntable, which will transport the inductor to the camera and trigger the light source to light and the camera to take a picture, obtaining the original image of the inductor to be measured, as shown in Figure 2 .

[0051] 2) Perform bilinear interpolation on the original image, enlarge the image scale to twice the original, and obtain a super scale image to enlarge the area of weak defects, so that the Faster-RCNN model can more easily find weak defects;

[0052] The process of bilinear interpolation is as follows:

[0053] f(x,y)=w(Q 11 )(x2-x)(y2-y)+w(Q 21 )(x-x1)(y2-y)+w(Q 12 )(x2-x)(y-y1)+w(Q 22 )(x-x1)(y-y1)

[0054]

[0055] In the formula, f(x,y) is the pixel value of the super scale image after interpolation at coordinates (x,y), x is the horizontal coordinate, y is the vertical coordinate, x1, x2, y1, y2 are the horizontal and vertical coordinates of the nearest neighbor points, Q ij is the four nearest neighbor points around the pixel point to be interpolated, and f(Q ij ) is the pixel value of the four nearest neighbor points. x i and y j represent the horizontal and vertical coordinates of the pixel respectively. W(Q ij ) represents the weight of different nearest neighbor points, and the smaller the distance from the predicted point, the greater the weight, wherein w(Q 11 ), w(Q 12 ), w(Q 21 ), w(Q 22 ) are the weights of the upper left, lower left, upper right and lower right pixel points of the pixel to be interpolated, and the super scale image is as shown in Figure 3 .

[0056] 3) Based on Faster-RCNN model, using Resnet50 as feature extraction module, extracting the features of original image and super scale image respectively; using Resnet50 to extract the features of original image, obtaining the original image features containing rich semantic information; in order to retain the detail information in super scale image which is beneficial to defect detection, only using the first three layers of Resnet50 to extract the features of super scale image, obtaining the super scale image features containing shallow detail information; Resnet50 is composed of convolution layer, pooling layer and nonlinear mapping;

[0057] The convolution process is as follows:

[0058]

[0059] In the formula, f1[x,y] is the data of the image in the (x,y) region, w[x,y] is the convolution kernel, f2[x,y] is the feature obtained after convolution, n i ,n j represents the offset distance from the convolution center, n1 and n2 are the maximum offset distances in the vertical and horizontal directions respectively, f[x+n i ,y+n j ] is the value of the image in (x+n i ,y+n j ), w[n i ,n j ] is the weight of the convolution kernel in the position (n i ,n j );

[0060] The nonlinear mapping process is as follows:

[0061] f3[x,y]=max(0,f2[x,y])

[0062] In the formula, f3[x,y] is the feature map obtained after nonlinear mapping.

[0063] 4) Because the super scale image features obtained in the previous step have a large scale, they need to be scaled to the same size as the original image features, so maximum pooling is used to retain important information in the feature map and remove redundant information.

[0064] 5) The original image features and the scaled super scale image features are spliced according to the channel, and then a 1*1 convolution kernel is used to transform the channel number while keeping the size of the feature map unchanged, so as to promote the information fusion between channels.

[0065] Wherein, the calculation of the size of the feature map after convolution is as follows:

[0066]

[0067] In the formula, Wout Wout = (W in - F + 2 * padding) / stride + 1 in Wout = (W in - F + 2 * padding) / stride + 1

[0068] 6) The fused feature map in the last step is input into the detection head of the Faster-RCNN model to locate and classify the component defects and output the defect detection information.

[0069] The task of the detection head includes two tasks of regression and classification, and is performed in two stages. The first stage is the binary classification of foreground and background and the regression of the foreground frame, that is, judging whether the anchor frame contains defects, and performing regression on the anchor frame containing defects to make the position of the anchor frame more accurate. The second stage is the classification of the foreground frame and the second regression of the foreground frame. In this embodiment, the classification task of the second stage is to classify the types of defects in the frame, and common inductor defect types are scratches, stains, etc. The regression task is to make the position of the foreground frame more accurate to better cover the defects.

[0070] The classification task uses a cross-entropy loss function, which has the following form:

[0071]

[0072] In the formula, L cls is the classification loss, N is the total number of samples, M is the total number of categories, y ic is a symbol function, c represents a category, and i represents a sample. If the real category of sample i is c, then y ic = 1, otherwise 0, P ic represents the predicted probability that sample i belongs to category c.

[0073] The regression task uses a smooth L1 loss, which is as follows:

[0074]

[0075] In the formula, smooth L1 represents the total loss, and l represents the loss value of each sample.

[0076] The final prediction result is as shown in Figure 4 .

[0077] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A high-precision detection method for weak defects in electronic components based on the Faster-RCNN model, characterized in that, Includes the following steps: 1) Acquire the original image of the component to be tested; 2) Perform bilinear interpolation on the original image from step 1) to obtain a hyperscale image that is twice the size of the original image; 3) The original image is used as the first-order input of the Faster-RCNN model. After passing through the entire feature extraction module, feature extraction is performed to obtain the original image features containing high-level semantic information. The superscale image is used as the input to the second stream of the Faster-RCNN model. After passing through the first three layers of the feature extraction module, feature extraction is performed to retain the detailed information in the superscale image that is beneficial for weak defect detection, and the superscale image features are obtained. Based on Faster-RCNN, ResNet50 is used as the feature extraction module to extract features from the original image and the superscale image respectively. ResNet50 is used to extract features from the original image to obtain original image features containing rich semantic information. In order to preserve the detailed information in the superscale image that is beneficial for defect detection, only the first three layers of ResNet50 are used to extract features from the superscale image to obtain superscale image features containing shallow detailed information. ResNet50 consists of convolutional layers, pooling layers, and nonlinear mappings. The convolution process is as follows: In the formula, f1[x,y] represents the image data in the (x,y) region, w[x,y] is the convolution kernel, f2[x,y] is the feature obtained after convolution, and n i n j f[x+n] represents the offset distance from the center of the convolution, n1 and n2 are the maximum offset distances in the vertical and horizontal directions of the convolution, respectively. i ,y+n j ] is the image in (x+n i ,y+n j The value of w[n) i ,n j ] is the convolution kernel in (n i ,n j The weight of the position; Its nonlinear mapping process: f3[x,y]=max(0,f2[x,y]) In the formula, f3[x,y] is the feature map obtained after performing nonlinear mapping; 4) Downsample the features of the superscale image to match the feature size of the original image; 5) Concatenate the superscale map features obtained in the previous step with the original map features, and use a 1*1 convolution kernel to perform channel transformation and fusion on the concatenated feature map; 6) Input the feature map fused in the previous step into the detection head of the Faster-RCNN model to locate and classify component defects, and output defect detection information.

2. The high-precision detection method for weak defects in components based on the Faster-RCNN model according to claim 1, characterized in that, In step 1), the component to be tested is placed in a tray. After the tray is shaken, the component is placed in an orderly manner on a transparent disc. The disc delivers the component to be tested to the camera. Hardware triggering is used to control the light source and the camera to take pictures, so as to obtain the original image of the component to be tested.

3. The high-precision detection method for weak defects in components based on the Faster-RCNN model according to claim 2, characterized in that, In step 2), bilinear interpolation is performed on the original image to enlarge the image scale to twice its original size, resulting in a superscale image, which enlarges the area of ​​weak defects, making it easier for the Faster-RCNN model to detect weak defects. The process of bilinear interpolation is as follows: f(x,y)=w(Q 11 )(x2-x)(y2-y)+w(Q 21 )(x-x1)(y2-y)+w(Q 12 )(x2-x)(y-y1)+ w(Q 22 )(x-x1)(y-y1) In the formula, f(x,y) is the pixel value of the interpolated hyperscale image at coordinates (x,y), where x is the horizontal coordinate, y is the vertical coordinate, and x1, x2, y1, y2 are the horizontal and vertical coordinates of the nearest neighbors. Q ij The four nearest neighbors of the pixel to be interpolated, f(Q) ij ) represents the pixel values ​​of the four nearest neighbors, x i and y j W(Q) represents the x and y coordinates of a pixel, respectively; ij ) represents the weight of different nearest neighbors. The smaller the distance to the point to be predicted, the greater the weight. Here, w(Q) 11 w(Q) 12 w(Q) 21 w(Q) 22 ) represent the weights of the top-left, bottom-left, top-right, and bottom-right pixels of the pixel to be interpolated.

4. The high-precision detection method for weak defects in components based on the Faster-RCNN model according to claim 3, characterized in that, In step 4), since the superscale map feature scale obtained in the previous step is large, it needs to be scaled down to the same size as the original map feature. Therefore, max pooling is used to retain important information in the feature map and remove redundant information.

5. The high-precision detection method for weak defects in components based on the Faster-RCNN model according to claim 4, characterized in that, In step 5), the original image features and the scaled-down superscale image features are concatenated by channel, and then a 1*1 convolution kernel is used to change the number of channels while keeping the feature map size unchanged, thereby promoting information fusion between channels. The size of the feature map after convolution is calculated as follows: In the formula, W out W represents the size of the output feature map. in The input feature map is denoted by padding, the number of padding loops is denoted by F, the kernel size is denoted by stride, and the output feature map is denoted by stride. When padding is 0, stride is 1, and the kernel size is 1, the output feature map size remains unchanged, and the number of channels in the output feature map is determined by the number of 1*1 kernels.

6. The high-precision detection method for weak defects in components based on the Faster-RCNN model according to claim 5, characterized in that, In step 6), the feature map fused in the previous step is input into the detector head of the Faster-RCNN model. The detector head has two tasks: regression and classification, which are performed in two stages. The first stage is the binary classification of foreground and background and the regression of foreground boxes, that is, determining whether the anchor box contains a defect and regressing the anchor boxes containing defects to make the anchor box position more accurate. The second stage is the classification of foreground boxes and the second regression of foreground boxes, which classifies the types of defects in the boxes. The regression task is to make the position of the foreground box more accurate and better cover the defects. The classification task uses the cross-entropy loss function, which takes the following form: In the formula, L cls For classification loss, N is the total number of samples, M is the total number of classes, and y ic It is a sign function, where c represents the class and i represents the sample. If the true class of sample i is c, then y ic =1, otherwise 0, P ic This represents the predicted probability that observed sample i belongs to category c; The regression task uses smooth L1 loss, as follows: In the formula, smooth L1 represents the total loss, and l represents the loss value for each sample.

Images