Industrial product defect detection method based on improved Faster RCNN
By improving the FasterRCNN algorithm, combined with the BS-Star module and the SA-NET module, the problem of low detection accuracy of small and medium-sized targets for industrial product defect detection is solved, and efficient detection of large, medium and small targets is achieved, and detection efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202411415637.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-11-29
AI Technical Summary
The existing industrial product defect detection technology has low accuracy for small targets in complex contexts, and it is difficult to take into account the detection of large, medium and small targets at the same time. Traditional machine recognition algorithms require manual processing, resulting in the inability to ensure detection efficiency and accuracy.
Improve the FasterRCNN algorithm, replace the convolution module of ResNet18 into the BS-Star module, add the hybrid attention mechanism SA-NET module, and improve the feature fusion pyramid network MFPN to enhance the feature extraction capability of the model and adapt to large, medium and small targets. Test.
It improves detection accuracy and speed, can accurately detect small targets in complex backgrounds, enhances feature extraction capabilities, realizes the classification, detection and segmentation of industrial product defects, and improves detection efficiency.
Smart Images

Figure CN119048487A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer vision and relates to an industrial product defect detection method based on an improved FasterRCNN. Background Art
[0002] In the process of industrial production, some industrial parts will inevitably have some tiny surface defects due to production technology and other issues. Therefore, in the detection of industrial parts defects, accurate target detection technology is needed to locate small defective industrial parts. However, the existing detection technology is limited by the fact that manual detection is very laborious; traditional machine recognition algorithms also need to use manual methods to process images. In the actual production process, it is often difficult to achieve the expected results of the machine vision defect detection system, so that neither the accuracy nor the speed of detection can be guaranteed.
[0003] At present, the defect detection network based on supervision includes: two-stage RCNN series detection algorithm, single-stage YOLO series detection algorithm, although compared with the RCNN series detection algorithm, the YOLO series detection algorithm is more efficient and faster, but its limitation is that when detecting smaller targets, because the single grid unit of YOLO can regress two bounding boxes of the same type at the same time, but its limitation is that the accuracy is not high when detecting small targets, and in the actual manufacturing process of industrial products, there are many types of defects, and the predictability of defect formation is poor and irregular. Therefore, the detection accuracy of YOLO will be lower than that of the RCNN series. The present invention will use the improved FasterRCNN algorithm to detect industrial products, by improving the Backbone module backbone network of the FasterRCNN algorithm, and at the same time improving the FPN of the Backbone module into a multi-information feature fusion pyramid network (MFPN), and adding a hybrid attention mechanism SA-NET module to the Backbone module to improve the feature extraction ability of the model, so as to achieve the purpose of improving detection accuracy and speed under complex backgrounds, and at the same time taking into account the detection of large targets, medium targets and small targets. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide an industrial product defect detection method based on improved FasterRCNN. This method is based on the original FasterRCNN. After improving the ResNet18 network and adding the SA-NET module, it can adapt to large targets, medium targets and small targets, realize the classification, detection and segmentation of defects, and improve the efficiency and speed of detection by improving the Backbone module.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] A technical solution for industrial product defect detection based on improved FasterRCNN:
[0007] (1) The improved residual network ResNet18 and the improved feature fusion pyramid network (FPN) are used as the backbone network. The residual network ResNet18 contains 4 convolution modules. In the improved residual network ResNet18, the convolution module is replaced by the BS-Star module. The BS-Star module uses 3×3 small convolutions instead of the original 7×7 large convolutions. Because the use of 7×7 large convolutions has a large number of parameters, it will also destroy the information and detailed texture of the original image, resulting in poor performance of subsequent network learning. The BS-Star module replaces the 7×7 large convolution with a small convolution to reduce information loss and parameter loss;
[0008] (2) The industrial product defect detection method based on the improved Faster RCNN according to claim 1 is characterized in that: the BS-Star module includes a 3×3 convolution layer through a normalization layer, and a deep convolution fusion Star module is added, and the Star module is expressed as:
[0009]
[0010] Among them, W1 is the transformation matrix of the output channel, W2 is the single-element input matrix of the output channel, X represents the input eigenvector, W1, is the dimension of the feature space, d is the number of input channels, T is the transpose operation of the matrix, are the elements in W1 and W2, is the element in the input feature vector X.
[0011] (3) The industrial product defect detection method based on the improved Faster RCNN according to claim 1 is characterized in that: the improved feature fusion pyramid network MFPN adopts a bidirectional structure, and the network structure of ResNet18 can be divided into four stages, each stage is composed of different residual blocks. In the first stage, the input image passes through a 3×3 convolution layer and a 3×3 maximum pooling layer, the size is reduced from 112×112 to 56×56, and the number of channels is increased from 3 to 64. The feature map obtained is C4. In the second stage, the BS-Star module is used to downsample the image using a convolution with a step size of 2. The feature map obtained by reducing the size of the image from 56x56 to 28x28 and increasing the number of channels from 64 to 128 is C3. In the third stage, the BS-Star module is used again to downsample the image size from 28x28 to 14x14 and increase the number of channels from 128 to 256. The feature map obtained is C2. In the fourth stage, the BS-Star module is used to downsample the image size from 14x14 to 7x7 and increase the number of channels from 256 to 512. The feature map obtained is C1. The top-down and horizontally connected feature maps of the MFPN network are fused to P1, P2, P3, and P4, respectively, and a lightweight upsampling module DySample is added between C4, P3, P2, and P1 to achieve multi-scale feature fusion.
[0012] (4) The industrial product defect detection method based on the improved Faster RCNN according to claim 1 is characterized in that: the hybrid attention mechanism SA-NET module combines the attention mechanisms of the feature channel and the feature space in two dimensions, the channel attention mechanism applies attention weights on the image feature channel scale, the spatial attention mechanism applies attention weights on the image feature space scale, and the hybrid attention mechanism SA-NET module combines the feature channel and the feature space in two dimensions;
[0013] The SA module is divided into three parts: feature grouping, mixed attention and feature aggregation. The input feature map is divided into multiple groups. The feature map is represented as X = [X1,...,X G ], is the dimension of the feature space, G is the number of groups, k is the index of the feature map, X G is the number of groups of feature graphs, X k is the grouping of feature maps, C is the number of channels, H is the height, W is the width, and C / G×H×W is the feature map X k Dimensions
[0014] The SA-NET module uses the following single-layer transformation formula:
[0015]
[0016]
[0017] X′ k2 =σ(W2·GN(X k2 )+b2)·X k2
[0018] Among them, s is the vector after global feature average pooling, is the average pooling feature map X k1 , X k1 is the input feature map, X k1 (i,j) is X k1 The element at position (i, j) in , σ is the Sigmoid activation function, is the linear transformation of the global feature s vector, To convert the global feature vector s into attention weights through the Sigmoid function, X′ k1 , X′ k2 For X k1 The sub-feature map, H×W is the spatial dimension of the feature map, GN(X k2 ) is for X′ k2 Normalized, W1, W2 are X′ k1 , X′ k2 The weight matrix of W1, b1 and b2 are the partial matrices of W1 and W2;
[0019] The image is output after classification and regression through the PRN (Region Proposal Network) module and the ROIPooling (Region of Interest Pooling) module. The regression loss of ROIPooling adopts the Smooth L1 loss function. The mathematical definition of the Smooth L1 loss function is as follows:
[0020]
[0021] Among them, x is the predicted value, y is the target value, and when the absolute value of the difference between the predicted value and the target value is less than 1, the loss function is expressed as a square term.
[0022] (5) The industrial product defect detection method based on the improved Faster RCNN according to claim 1 is characterized in that: the upsampling method used for the low-resolution feature map C4 generated in the first stage of the ResNet18 network is PixelShuffle, and the Pixel Shuffle upsampling is used at low resolution without losing detail information, and the total amount of information remains unchanged;
[0023] The upsampling method used by the ResNet18 network to generate feature maps C1, C2, and C3 of different resolutions is bilinear interpolation. Bilinear interpolation only uses four neighbor neurons with integer coordinates around neuron S. They are usually the neurons in the upper left, upper right, lower left, and lower right corners of neuron S to calculate f(s). f(s) represents the feature values of these four specific positions. Assume that the position coordinates of these four neurons are (x0, y0), (x1, y0), (x0, y1), and (x1, y1), and their values are f(x0, y0), f(x1, y0), f(x0, y1), and f(x1, y1), respectively. For neuron S at position S(x, y), the specific formula for calculating its value f(s) using bilinear interpolation is as follows:
[0024] f(s)=(1-v)·R1+v·R2
[0025] R1=(1-u)·f(x0,y0 ) +u·f(x1,y0)
[0026] R2=(1-u)·f(x0, y1)+u·f(x1, y1)
[0027] u=x-x0
[0028] v=y-y0
[0029] where u and v are the ratios of the horizontal and vertical distances of neuron S relative to (x0, y0).
[0030] (6) The industrial product defect detection method based on the improved Faster RCNN according to claim 1 is characterized in that the loss function of the improved Faster RCNN is:
[0031] L Loss =L rcls +L rbbox +L cls +L bbox+Lsenet
[0032] Among them, L rcls is the classification loss of the region proposal network, L rbbox is the bounding box regression loss of the region proposal network, L cls is the classification loss of the entire network, L bbox is the bounding box regression loss of the entire network, L senet The loss of feature extraction for the SA-NET module;
[0033]
[0034] Among them, α is the weight coefficient for measuring positive and negative samples, γ is the weight coefficient for adjusting the difference between simple samples and difficult samples, m is the category parameter, positive samples are 1, negative samples are 0, K is the number of categories, and P is the probability that the target belongs to a certain category.
[0035]
[0036] in, is the center coordinate of the rectangular box, are the width and height of the rectangular box respectively, Smooth L1 is the regression loss function of ROIPooling, is the regression parameter of the true box; is the regression parameter of the prediction box. C is the number of categories, is an indicator variable, which is 1 if category C is the correct category of sample i, otherwise it is 0; is the probability that the model predicts that sample i is of category C;
[0037] The beneficial effects of the present invention are:
[0038] (1) The purpose of the present invention is to provide an industrial product defect detection method based on an improved Faster RCNN, which solves the problem of using traditional visual methods to detect defects with complex texture backgrounds and small target defects in industrial product defect detection. It also overcomes the problem that it is difficult to take into account the detection accuracy of large targets, medium targets and small targets at the same time. At the same time, the network is also improved so that the network can have a self-attention mechanism weighting between image features, increase the ability to extract image features and strengthen the connection between images.
[0039] (2) The present invention adopts the improved Faster RCNN as the main detection algorithm, takes the improved ResNet18+MFPN as the backbone network, and fuses the 4-layer feature map output by ResNet18 with the improved feature pyramid network (MFPN). The improved MFPN abandons the original one-to-one correspondence principle of feature map fusion, and adopts a feature map to fuse with the corresponding 4 feature maps of FPN, which can realize the extraction of multi-scale feature maps.
[0040] (3) The present invention adds a hybrid attention mechanism SA-NET module, which can fully combine the attention mechanisms of the feature channel and feature space. The channel attention mechanism is to apply attention weights on the image feature channel scale, so that the model pays attention to meaningful features; the spatial attention mechanism is to apply attention weights on the image feature space scale, so that the model pays attention to the specific position of the feature in space; the hybrid attention mechanism SA-NET module can fully combine the attention mechanisms of the feature channel and feature space in two dimensions, and comprehensively analyze the image features so that the model can accurately pay attention to the target features in the image. In order to better analyze the fused image features, a more accurate pre-selected box is generated in the region proposal network.
[0041] (4) The present invention achieves mapping of high-dimensional feature space by adding BS-Star module through star operation while keeping the computational complexity low. This solves the trade-off between computational complexity and performance in traditional efficient network design. In target detection tasks, especially for small targets and complex scenes, rich feature representation can significantly improve the detection effect, making the RCNN series network perform better in these scenes. Star operation provides a method for simplifying network design, which can achieve efficient feature representation without complex feature fusion and multi-branch design.
[0042] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0044] Figure 1 The figure is a flow chart of an industrial product defect detection method based on improved Faster RCNN of the present invention.
[0045] Figure 2 This is a diagram of the process of defect detection and defect location for an image based on an improved Faster RCNN classification detection framework according to the present invention.
[0046] Figure 3 It is the structural diagram of the BS-Star module of the present invention. DETAILED DESCRIPTION
[0047] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.
[0048] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0049] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0050] like Figure 1 As shown, the present invention provides a flow chart of an industrial product defect detection method based on an improved Faster RCNN, comprising the following specific steps:
[0051] (1) Obtain a data set, preprocess the data set, and then classify the self-made data set;
[0052] (2) Perform data enhancement and expansion on the preprocessed dataset, and use the image annotation tool LabelImg to annotate the dataset and convert it into a unified format;
[0053] (3) The improved Faster RCNN is used as the main algorithm for target detection, and the improved ResNet18+MFPN replaces the original ResNet18+FPN as the backbone network;
[0054] (4) Add the hybrid attention mechanism SA-NET to the ResNet18+MFPN module;
[0055] (5) Output the feature maps extracted in step (3) to the hybrid attention mechanism SA-NET module respectively;
[0056] (6) The feature map obtained in step (5) is classified and regressed through the PRN module and ROI module and then output.
[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. An industrial product defect detection method based on improved Faster RCNN, characterized by: The method comprises the following steps: Obtain a dataset containing images of industrial product defects; Preprocess the acquired data set, including image scaling, cropping, and normalization; Perform data augmentation on the preprocessed data set, including rotating, flipping, and cropping the image to expand the data set; Use image annotation tools to annotate the data set and mark the defect area as the region of interest (ROI); A defect detection model based on an improved faster region convolutional neural network (Faster Region Proposal Network, FasterRCNN) is constructed, the model comprising: The improved ResNet18 network and the improved Multi-Feature Pyramid Networks (MFPN) are used as the backbone network, in which the convolution module of the ResNet18 network is replaced by the Basic Bolck_Star (BS-Star) module, and a lightweight upsampling module DySample is added to the MFPN to achieve multi-scale feature fusion; A hybrid attention mechanism (SEAttation-Networks, SA-NET) module is added after the backbone network to fully combine the attention mechanisms of the feature channel and feature space dimensions to improve the feature extraction capability of the model; Use the labeled data set to train the constructed model and optimize the model parameters so that the model can accurately identify and locate industrial product defects; Use the trained model to perform defect detection on the input industrial product images and output the defect category, location and segmentation results.
2. The industrial product defect detection method based on improved Faster RCNN according to claim 1, characterized in that: The BS-Star module contains a 3×3 convolution layer through a normalization layer, and a deep convolution fusion Star module is added. The Star module is expressed as: Among them, W1 is the transformation matrix of the output channel, W2 is the single-element input matrix of the output channel, X represents the input eigenvector, is the dimension of the feature space, d is the number of input channels, T is the transpose operation of the matrix, are the elements in W1 and W2, is the element in the input feature vector X.
3. The industrial product defect detection method based on improved Faster RCNN according to claim 1, characterized in that: The improved feature fusion pyramid network MFPN adopts a bidirectional structure. The network structure of ResNet18 can be divided into four stages, each of which is composed of different residual blocks. In the first stage, the input image passes through a 3×3 convolution layer and a 3×3 maximum pooling layer, the size is reduced from 112×112 to 56×56, and the number of channels is increased from 3 to 64 to obtain the feature map C4. In the second stage, through the BS-Star module, a convolution with a step size of 2 is used for downsampling, the image size is reduced from 56x56 to 28x28, and the number of channels is increased from 64 to 128 to obtain the feature map C3. In the third stage, the BS-Star module is used again. In the fourth stage, the convolution with a stride of 2 is used for downsampling again, and the image size is reduced from 28x28 to 14x14, and the number of channels is increased from 128 to 256. The feature map obtained is C2. In the fourth stage, the BS-Star module is finally used to downsample the image size from 14x14 to 7x7, and the number of channels is increased from 256 to 512. The feature map obtained is C1; the feature maps of the top-down and horizontal connections of the MFPN network are fused to P1, P2, P3, and P4 respectively, and a lightweight upsampling module DySample is added between C4, P3, P2, and P1 to achieve multi-scale feature fusion.
4. The industrial product defect detection method based on improved Faster RCNN according to claim 1, characterized in that: The hybrid attention mechanism SA-NET module combines the attention mechanisms of the feature channel and the feature space in two dimensions. The channel attention mechanism applies attention weights on the image feature channel scale, and the spatial attention mechanism applies attention weights on the image feature space scale. The hybrid attention mechanism SA-NET module combines the feature channel and the feature space in two dimensions. The SA module is divided into three parts: feature grouping, mixed attention and feature aggregation. The input feature map is divided into multiple groups. The feature map is represented as is the dimension of the feature space, G is the number of groups, k is the index of the feature map, X G is the number of groups of feature graphs, X k is the grouping of feature maps, C is the number of channels, H is the height, W is the width, and C / G×H×W is the feature map X k Dimensions The SA-NET module uses the following single-layer transformation formula: X’ k2 =σ(W2·GN(X k2 )+b2)·X k2 Among them, s is the vector after global feature average pooling, is the average pooling feature map X k1 , X k1 is the input feature map, X k1 (i,j) is X k1 The element at position (i, j) in , σ is the Sigmoid activation function, is the linear transformation of the global feature s vector, To convert the global feature vector s into attention weights through the Sigmoid function, X' k1 , X' k2 For X k1 The sub-feature map, H×W is the spatial dimension of the feature map, GN(X k2 ) is for X' k2 Normalized, W1, W2 are X' k1 , X' k2 The weight matrix of W1, b1 and b2 are the partial matrices of W1 and W2; The image is output after classification and regression through the Region Proposal Network (PRN) module and the Region of Interest Pooling (ROI Pooling) module. The regression loss of ROI Pooling adopts the Smooth L1 loss function. The mathematical definition of the Smooth L1 loss function is as follows: Among them, x is the predicted value, y is the target value, and when the absolute value of the difference between the predicted value and the target value is less than 1, the loss function is expressed as a square term.
5. The industrial product defect detection method based on improved Faster RCNN according to claim 1, characterized in that: The upsampling method used for the low-resolution feature map C4 generated in the first stage of the ResNet18 network is PixelShuffle. When the resolution is low, PixelShuffle upsampling is used without losing detail information, and the total amount of information remains unchanged. The upsampling method used by the ResNet18 network to generate feature maps C1, C2, and C3 of different resolutions is bilinear interpolation. Bilinear interpolation only uses four neighbor neurons with integer coordinates around neuron S. They are usually the neurons in the upper left, upper right, lower left, and lower right corners of neuron S to calculate f(s). f(s) represents the feature values of these four specific positions. Assume that the position coordinates of these four neurons are (x0, y0), (x1, y0), (x0, y1), and (x1, y1), and their values are f(x0, y0), f(x1, y0), f(x0, y1), and f(x1, y1), respectively. For neuron S at position S(x, y), the specific formula for calculating its value f(s) using bilinear interpolation is as follows: f(s)=(1-v)·R1+v·R2 R1=(1-u)·f(x0,y0)+u·f(x1,y0) R2=(1-u)·f(x0, y1)+u·f(x1, y1) u=x-x0 v=y-y0 where u and v are the ratios of the horizontal and vertical distances of neuron S relative to (x0, y0).
6. The industrial product defect detection method based on improved Faster RCNN according to claim 1, characterized in that: The loss function of the improved Faster RCNN is: L Loss =L rcls +L rbbox +L cls +L bbox +L senet Among them, L rcls is the classification loss of the region proposal network, L rbbox is the bounding box regression loss of the region proposal network, L cls is the classification loss of the entire network, L bbox is the bounding box regression loss of the entire network, L senet The loss of feature extraction for the SA-NET module; Among them, α is the weight coefficient for measuring positive and negative samples, γ is the weight coefficient for adjusting the difference between simple samples and difficult samples, m is the category parameter, positive samples are 1, negative samples are 0, K is the number of categories, and P is the probability that the target belongs to a certain category. in, is the center coordinate of the rectangular box, are the width and height of the rectangular box respectively, Smooth L1 is the regression loss function of ROIPooling, is the regression parameter of the true box; is the regression parameter of the prediction box; C is the number of categories, is an indicator variable, which is 1 if category C is the correct category of sample i, otherwise it is 0; is the probability that the model predicts that sample i is of category C.
Citation Information
Cited By
Rolling metal surface defect automatic labeling method based on multi-task self-adaptive model
CN119444759A
Improved agricultural pest target detection method based on mixed attention mechanism
CN119963824A
Logistics sorting algorithm based on RCNN
CN120198734A
Confusion expression recognition method and device fusing lightweight double-branch attention, equipment and storage medium
CN121074966A
Passenger car part anomaly detection method based on deep residual network
CN122223009A
