Copper bush appearance defect detection method and system based on fusion attention mechanism

Abstract

The invention discloses a copper bush appearance defect detection method and system based on a fusion attention mechanism, and the method comprises the steps: building a copper bush appearance defect detection model, collecting image data containing copper bush appearance defects, manufacturing a copper bush appearance defect data set, and dividing the copper bush appearance defect data set into a training set, a verification set and a test set; training, verifying and testing the copper bush appearance defect detection model to obtain a tested copper bush appearance defect detection model, acquiring an RGB image of a to-be-detected copper bush from a copper bush production line in real time, performing data enhancement processing, and inputting the data enhanced image into the tested copper bush appearance defect detection model for processing to obtain the copper bush appearance defect detection model. Outputting the detection target of the to-be-detected copper bush, and obtaining an appearance defect detection result of the to-be-detected copper bush according to the detection target of the to-be-detected copper bush. According to the method, the copper bush appearance defect detection efficiency can be greatly improved, and the migration and generalization ability of a copper bush appearance defect detection model is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a copper sleeve appearance defect detection method and system based on a fusion attention mechanism. Background Art

[0002] In the process of producing copper sleeves on industrial production lines, appearance defects usually include too long tail wires, missing copper sheets, skewed sticks, and misaligned copper sheets. Such defects will seriously affect the quality of the copper sleeves and further affect the performance of the entire equipment. Therefore, real-time and efficient detection of copper sleeve appearance defects is crucial to ensure product quality.

[0003] Most existing defect detection methods use deep learning technology, and mainstream target detection algorithms are mainly divided into two-stage and one-stage. Although two-stage algorithms such as Fast-RCNN have superior detection performance, they have high computational complexity and are not suitable for real-time applications. One-stage algorithms such as SSD and YOLO series have significantly improved detection speed, especially the YOLO series due to its simple architecture and strong real-time performance, which is widely used in industrial detection. However, these algorithms perform poorly when dealing with small targets or irregularly shaped defects, and it is difficult to balance detection accuracy and real-time performance.

[0004] In response to the above problems, some researchers have made a number of improvements to the YOLO series of algorithms. For example, YOLO9000 improves detection speed and accuracy through multi-scale processing and anchor box technology, but still has insufficient performance on small targets; RA-YOLO improves the detection performance of small targets by introducing scale perception and scale perception mechanisms, but the computational overhead increases; YOLOF improves detection accuracy by using a single-scale output decoder, but has limitations in multi-scale target detection; YOLOv6 and YOLOv7 further improve detection efficiency and accuracy by optimizing the backbone network and introducing a new detector architecture, but these improvements still have limited generalization capabilities in complex scenarios.

[0005] In order to improve detection performance in an environment with limited computing resources, lightweight networks such as MobileNet and GhostNet have emerged. They reduce the model size and improve the reasoning speed by cropping, grouping convolution, etc. However, these methods lack the correlation between feature maps of different dimensions, resulting in limited detection accuracy. In addition, although the introduction of attention mechanisms (such as CBAM, SE modules, FcaNet) has improved the detection capabilities of small targets and complex scenes, when dealing with defects of small size and irregular shape, existing methods still cannot achieve ideal accuracy while ensuring real-time performance.

[0006] In summary, the existing technology has not yet achieved an ideal balance between real-time performance and accuracy in copper sleeve appearance defect detection, especially when facing defects of small size and complex shape, the detection capability still needs to be improved. The present invention aims to solve these technical problems and optimize the real-time performance and accuracy of the detection algorithm to better meet the needs of industrial production lines. Summary of the invention

[0007] In response to the above problems, the present invention application proposes a copper sleeve appearance defect detection method and system based on a fusion attention mechanism to effectively improve the accuracy and efficiency of copper sleeve appearance defect detection.

[0008] A copper sleeve appearance defect detection method based on a fusion attention mechanism comprises the following steps:

[0009] S1. Based on the YOLOv5 target detection network, a copper sleeve appearance defect detection model is built. The model includes an improved backbone network, an improved bottleneck layer and a prediction layer connected in sequence. The improved backbone network is used to extract features from the input image, the improved bottleneck layer is used to receive and process the feature map extracted by the backbone network, and output the spatial attention feature map. The prediction layer is used to predict the spatial attention feature map to obtain the detection target corresponding to the input image.

[0010] S2, collecting image data containing copper sleeve appearance defects and creating a copper sleeve appearance defect data set, dividing the copper sleeve appearance defect data set into a training set, a validation set and a test set, inputting the training set into a copper sleeve appearance defect detection model for training, and using the validation set and the test set to verify and test the trained model, and reversely updating the model weights and parameters to obtain a copper sleeve appearance defect detection model after the test;

[0011] S3, acquiring the RGB image of the copper sleeve to be detected from the copper sleeve production line in real time and performing data enhancement processing to obtain an image with increased data of the copper sleeve to be detected;

[0012] S4. Input the image after data enhancement of the copper sleeve to be detected into the copper sleeve appearance defect detection model after the test, output the detection target of the copper sleeve to be detected, and obtain the appearance defect detection result of the copper sleeve to be detected according to the detection target of the copper sleeve to be detected. The appearance defect detection result includes defect category and confidence.

[0013] Preferably, the improved backbone network in S1 includes a Conv_BN module, a first bneck module introducing the SE attention mechanism, a second bneck module introducing the SE attention mechanism, and third to eleventh bneck modules connected in sequence. The image to be detected is input into the Conv_BN module for processing, and the feature map output by the Conv_BN module is processed and features are extracted respectively by the first bneck module introducing the SE attention mechanism, the second bneck module introducing the SE attention mechanism, and the third to eleventh bneck modules connected in sequence, and feature maps of different scales are output through the seventh bneck module, the tenth bneck module and the twelfth bneck module.

[0014] Preferably, each backneck module in the improved backbone network in S1 that introduces the SE attention mechanism extracts features from the input feature map, and the specific process is as follows:

[0015] S11. Each backneck module that introduces the SE attention mechanism receives the feature map input by the previous module adjacent to it, and generates convolution features through convolution operation;

[0016] S12, performing a global average pooling operation on the convolutional features, compressing the two-dimensional features of each channel, and obtaining a feature vector;

[0017] S13, processing the feature vector through two fully connected layers connected in sequence, normalizing the processed feature vector through an activation function, and obtaining a channel weight value;

[0018] S14. Weight the first feature map through the channel weight value to obtain the feature map output by each backneck module that introduces the SE attention mechanism.

[0019] Preferably, each bneck module in the improved backbone network in S1 includes several bneck blocks, each bneck block includes a deep packet convolution layer, a pooling layer and a fully connected activation function layer connected in sequence, and each bneck block uses a deep packet convolution method to extract features from the input feature map, specifically including the following steps:

[0020] S15, the deep grouped convolution layer of each bneck block receives the feature map output by the previous module adjacent to it, reduces the number of channels of the feature map by half, obtains the first feature and the second feature with half the number of channels, performs deep grouped convolution processing on the first feature with half the number of channels, obtains the third feature, and fuses and optimizes the third feature with the second feature with half the number of channels to form a dense feature map containing more appearance information;

[0021] S16, dividing the dense feature map into several rectangular regions and performing pooling processing through a pooling layer, reducing the dimension and width of each rectangular region to prevent overfitting, and obtaining a maximum region feature map;

[0022] S17, the fully connected activation function layer receives the maximum area feature map and converts it into a one-dimensional feature vector to obtain a feature map containing the target global features output by each bneck block.

[0023] Preferably, the improved bottleneck layer in S1 includes a first CBS module, a first Upsample module, a first Concat module, a first C3CBAM module, a second CBS module, a second Upsample module, a second Concat module, a second C3CBAM module, a third CBS module, a third Concat module, a third C3CBAM module, a fourth CBS module, a fourth Concat module, and a fourth C3CBAM module, which are connected in sequence. The first CBS module is also connected to the fourth Concat module, and the second CBS module is also connected to the third Concat module. The first CBS module, the first Concat module, and the third Concat module in the improved bottleneck layer respectively receive and process feature maps of different scales output by the improved backbone network, and output spatial attention feature maps of different scales through the second C3CBAM module, the third C3CBAM module, and the fourth C3CBAM module.

[0024] Preferably, each C3CBAM module in the improved bottleneck layer is obtained by integrating the CBAM mechanism into the C3 module. The CBAM mechanism includes a channel attention module and a spatial attention module connected in sequence. Each C3CBAM module in the improved bottleneck layer performs global maximum pooling and global average pooling of the spatial dimension on the input feature map, uses the channel attention module to correct the input feature map, obtains the channel attention feature map, performs global maximum pooling and global average pooling of the channel dimension on the channel attention feature map, and uses the spatial attention module to correct the channel attention feature map to obtain the spatial attention feature map. The specific formula is as follows:

[0025]

[0026]

[0027] Where F represents the feature map of each C3CBAM module input, Mc represents the one-dimensional convolution of the channel attention module, Ms represents the two-dimensional convolution of the spatial attention module, F′ represents the channel attention feature map, and F″ represents the spatial attention feature map.

[0028] Preferably, the prediction layer in S1 includes three parallel Conv convolution layers, and the three parallel Conv convolution layers are respectively connected to the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module in the improved bottleneck layer in a one-to-one correspondence. The three parallel Conv convolution layers respectively receive and process the spatial attention feature maps output by the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module, and output predicted small, medium and large detection targets.

[0029] Preferably, in S2, image data containing copper sleeve appearance defects are collected and a copper sleeve appearance defect data set is prepared, and the copper sleeve appearance defect data set is divided into a training set, a validation set and a test set. The specific process is as follows:

[0030] S21, using an industrial camera to collect and process images of the copper sleeve sample with appearance defects and the normal copper sleeve sample to obtain processed images;

[0031] S22, performing image enhancement processing and balancing processing on the pre-processed image to obtain an expanded image;

[0032] S23, using a Labeling tool to label the expanded image, and checking the accuracy and consistency of the labeled image, thereby obtaining a copper sleeve appearance defect data set including several types of appearances;

[0033] S24, dividing the copper sleeve appearance defect data set including several types of appearance into a training set, a verification set and a test set according to a preset ratio.

[0034] Preferably, the several types of appearance are: missing copper sheets, crooked sticks, misaligned copper sheets, too long tail wires, parallel copper sheets, and qualified.

[0035] On the other hand, the present invention provides a copper sleeve appearance defect detection system, which uses any of the above copper sleeve appearance defect detection methods based on the fusion attention mechanism for detection. The detection system includes: an image acquisition unit, an image processing unit, a copper sleeve appearance defect detection model and a computer system. The image acquisition unit is connected to the computer system, and the image processing unit is connected to the copper sleeve appearance defect detection model and is arranged in the computer system, wherein:

[0036] The image acquisition unit is used to collect image data of the copper sleeve to be inspected on the production line in real time, and send the collected image data to the computer system;

[0037] The computer system is used to receive the image data collected by the image collection unit and send it to the image processing unit;

[0038] The image processing unit is used to receive and process image data, and send the processed image data to the copper sleeve appearance defect detection model;

[0039] The copper sleeve appearance defect detection model is used to receive processed image data and perform feature extraction and prediction, and output the appearance defect detection results of the copper sleeve to be detected.

[0040] The above-mentioned copper sleeve appearance defect detection method and system based on the fusion attention mechanism builds a copper sleeve appearance defect detection model, which includes an improved backbone network, an improved bottleneck layer and a prediction layer connected in sequence, the improved backbone network is used to extract features from the input image, the improved bottleneck layer is used to receive and process the feature map extracted by the backbone network, output the spatial attention feature map, the prediction layer is used to predict the spatial attention feature map, obtain the detection target corresponding to the input image, collect image data containing copper sleeve appearance defects and make a copper sleeve appearance defect data set, divide the copper sleeve appearance defect data set into a training set, a verification set and a test set, and train, verify and test the copper sleeve appearance defect detection model, reversely update the model weights and parameters, obtain the tested copper sleeve appearance defect detection model, obtain the RGB image of the copper sleeve to be detected from the copper sleeve production line in real time and perform data enhancement processing, input the enhanced image of the copper sleeve data to be detected into the tested copper sleeve appearance defect detection model for processing, and output the appearance defect detection result of the copper sleeve to be detected. This method can greatly improve the efficiency of copper sleeve appearance defect detection and improve the migration and generalization ability of the copper sleeve appearance defect detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a copper sleeve appearance defect detection method based on a fusion attention mechanism in one embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the network structure of a copper sleeve appearance defect detection model in one embodiment of the present invention;

[0043] Figure 3 Schematic diagram of the working process of each bneck module that introduces the SE attention mechanism in one embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the working process of the bneck block in each bneck module in one embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of the working process of the C3CBAM module in the improved bottleneck layer in one embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the dimensions of a copper sleeve in one embodiment of the present invention;

[0047] Figure 7 6 types of appearance examples of copper sleeves in one embodiment of the present invention;

[0048] Figure 8 It is a schematic diagram of different positions of the backbone network after the attention mechanism is integrated and improved in one embodiment of the present invention;

[0049] Fig. 9 It is a comparison diagram of experimental effects under different schemes in one embodiment of the present invention;

[0050] Fig.10 This is a comparison diagram of the effects of various attention mechanisms in one embodiment of the present invention;

[0051] Fig.11 This is a heat map corresponding to six types of appearance defects analyzed by heat map visualization using a class activation mapping tool in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0053] In one embodiment, see Figure 1 and Figure 2 , a copper sleeve appearance defect detection method based on fusion attention mechanism, the method comprises the following steps:

[0054] S1. Based on the YOLOv5 target detection network, a copper sleeve appearance defect detection model is built. The model includes an improved backbone network, an improved bottleneck layer and a prediction layer connected in sequence. The improved backbone network is used to extract features from the input image, the improved bottleneck layer is used to receive and process the feature map extracted by the backbone network, and output the spatial attention feature map. The prediction layer is used to predict the spatial attention feature map to obtain the detection target corresponding to the input image.

[0055] Furthermore, the improved backbone network in S1 includes a Conv_BN module, a first bneck module introducing the SE attention mechanism, a second bneck module introducing the SE attention mechanism, and a third bneck module to an eleventh bneck module connected in sequence. The image to be detected is input into the Conv_BN module for processing, and the feature map output by the Conv_BN module is processed and features are extracted respectively through the first bneck module introducing the SE attention mechanism, the second bneck module introducing the SE attention mechanism, and the third bneck module to the eleventh bneck module connected in sequence, and feature maps of different scales are output through the seventh bneck module, the tenth bneck module and the twelfth bneck module.

[0056] Specifically, the YOLOv5 target detection network includes a backbone network, a bottleneck layer and a detection layer. The original backbone network in the YOLOv5 target detection network is specifically DarkNet-53. In order to enhance the recognition and classification capabilities of texture defects and structural defects in copper sleeves, the copper sleeve appearance defect detection model is based on the YOLOv5 target detection network. The original backbone network DarkNet-53 in the YOLOv5 target detection network is replaced with a lightweight network MobileNetV3S. The original lightweight network MobileNetV3S includes 16 stages. In this technical solution, by reducing the number of network layers, specifically removing stages 4, 7 and 9 in the original lightweight network MobileNetV3S, and then postponing the remaining stages, Thus, 13 stages are obtained. In order to further improve the migration and generalization ability of the copper sleeve appearance defect detection model, the SE attention mechanism is integrated into the second stage (corresponding to the first bneck module) and the third stage (corresponding to the second bneck module) of the lightweight network MobileNetV3S, which can expand the receptive field range and enhance the perception of subtle defect features. Thus, the improved backbone network is obtained. The image to be detected is input into the improved backbone network for processing. The modules corresponding to each stage extract feature maps of different scales from the input image in turn, and the first feature maps of different scales (size is 14) are output through the 7th, 10th, and 12th stages of the improved backbone network (corresponding to the sixth bneck module, the ninth bneck module, and the eleventh bneck module, respectively). 2 ×48), the second feature map (size is 7 2 ×512) and the third feature map (size 1 2 ×1024).

[0057] right Figure 2 The numbers in the bneck module are supplemented as follows:

[0058] Taking stage 8 (corresponding to the seventh bneck module) as an example: “×3” means that the bneck block is repeatedly stacked 3 times in the seventh bneck module, that is, the bneck block is used 3 times in succession in stage 8; 2 " refers to a 7x7 two-dimensional matrix, which represents the convolution kernel size of the convolution operation. 2 It indicates that the size of the convolution kernel is 7x7; "96" refers to the number of channels, that is, the depth of the feature map of this layer. 96 means that there are 96 feature channels in this layer, and the output of the convolution operation will be a feature map with 96 channels.

[0059] Furthermore, each backneck module in the improved backbone network in S1 that introduces the SE attention mechanism extracts features from the input feature map. The specific process is as follows:

[0060] S11. Each backneck module that introduces the SE attention mechanism receives the feature map input by the previous module adjacent to it, and generates convolution features through convolution operation;

[0061] S12, performing a global average pooling operation on the convolutional features, compressing the two-dimensional features of each channel, and obtaining a feature vector;

[0062] S13, processing the feature vector through two fully connected layers connected in sequence, normalizing the processed feature vector through an activation function, and obtaining a channel weight value;

[0063] S14. Weights are assigned to the convolution features through the channel weight values ​​to obtain the feature maps output by each bneck module that introduces the SE attention mechanism.

[0064] Specifically, see Figure 2 and Figure 3 , to introduce the second bneck module of SE attention mechanism (corresponding to Figure 2 The working process of stage 3) of the improved backbone network in the figure is illustrated by an example. The second bneck module that introduces the SE attention mechanism receives the low-dimensional feature map (denoted as X) output by the second bneck module that introduces the SE attention mechanism, generates a convolution feature U through a convolution operation, performs a global average pooling operation on the convolution feature U, and compresses the two-dimensional features (H×W) of each channel to obtain a 1×1×C feature vector, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map. The 1×1×C feature vector is processed by two sequentially connected fully connected layers, and the processed 1×1×C feature vector is normalized by an activation function to obtain a channel weight value, where the number of channel weight values ​​is the same as the number of channels of the input feature map, and the convolution feature U is weighted by the channel weight value to obtain a high-dimensional feature map output by the second bneck module that introduces the SE attention mechanism.

[0065] The correlation between channels is constructed through two fully connected layers. The input of the first fully connected layer of the two sequentially connected fully connected layers is a 1×1×C feature vector. In the first fully connected layer, the number of channels C is reduced to C / r layers, where r is the reduction ratio, to obtain the reduced feature vector. In the second fully connected layer, the reduced feature vector is restored to the original number of channels C, to obtain a 1×1×C feature vector processed by two sequentially connected fully connected layers.

[0066] Furthermore, each bneck module in the improved backbone network in S1 includes several bneck blocks, each of which includes a deep packet convolution layer, a pooling layer, and a fully connected activation function layer connected in sequence. Each bneck block uses a deep packet convolution method to extract features from the input feature map, specifically including the following steps:

[0067] S15, the deep grouped convolution layer of each bneck block receives the feature map output by the previous module adjacent to it, reduces the number of channels of the feature map by half, obtains the first feature and the second feature with half the number of channels, performs deep grouped convolution processing on the first feature with half the number of channels, obtains the third feature, and fuses and optimizes the third feature with the second feature with half the number of channels to form a dense feature map containing more appearance information;

[0068] S16, dividing the dense feature map into several rectangular regions and performing pooling processing through a pooling layer, reducing the dimension and width of each rectangular region to prevent overfitting, and obtaining a maximum region feature map;

[0069] S17, the fully connected activation function layer receives the maximum area feature map and converts it into a one-dimensional feature vector to obtain a feature map containing the target global features output by each bneck block.

[0070] Specifically, see Figure 2 and Figure 4 , with the third bneck module (corresponding to Figure 2 The third bneck module includes a bneck block, which includes a deep grouped convolutional layer, a pooling layer, and a fully connected activation function layer connected in sequence. At this time, the deep grouped convolutional layer receives the low-dimensional feature map (denoted as F1) output by the previous module adjacent to it (that is, the second bneck module introducing the SE attention mechanism), and reduces the number of channels of the low-dimensional feature map F1 by half, thereby obtaining the first feature map F with half the number of channels. 1 / 2 and the second feature F' 1 / 2 , the first feature F with half the number of channels 1 / 2 Perform deep group convolution to obtain the third feature F" 1 / 2 , the third feature F" 1 / 2 The second characteristic F' with half the number of channels 1 / 2Fusion optimization is performed to form a dense feature map F'2 containing more appearance information; the dense feature map F'2 is divided into several rectangular areas, and processed by the pooling layer of the bneck block to reduce the dimension and width of each rectangular area to prevent overfitting, and the maximum area feature map F'3 is obtained; the fully connected activation function layer of the bneck block receives the maximum area feature map F'3 and converts it into a one-dimensional feature vector, and the high-dimensional feature map F'4 containing the target global features output by the bneck block is obtained, that is, the high-dimensional feature map output by the third bneck module.

[0071] The bneck blocks in each bneck module in the improved backbone network use deep group convolution to extract features, which can greatly improve the efficiency of copper sleeve appearance defect detection. The bneck blocks can reduce the amount of calculation while retaining important feature information, thereby improving the expressiveness of the network.

[0072] Furthermore, the improved bottleneck layer in S1 includes a first CBS module, a first Upsample module, a first Concat module, a first C3CBAM module, a second CBS module, a second Upsample module, a second Concat module, a second C3CBAM module, a third CBS module, a third Concat module, a third C3CBAM module, a fourth CBS module, a fourth Concat module, and a fourth C3CBAM module, which are connected in sequence. The first CBS module is also connected to the fourth Concat module, and the second CBS module is also connected to the third Concat module. The first CBS module, the first Concat module, and the third Concat module in the improved bottleneck layer respectively receive and process feature maps of different scales output by the improved backbone network, and output spatial attention feature maps of different scales through the second C3CBAM module, the third C3CBAM module, and the fourth C3CBAM module.

[0073] Specifically, the improved bottleneck layer in the copper sleeve appearance defect detection model is based on the original bottleneck layer in the YOLOv5 target detection network. The CBAM attention mechanism is integrated into each C3 module in the original bottleneck layer, thereby obtaining the C3CBAM module to enhance the bottleneck layer's ability to integrate target features.

[0074] Figure 2 Each CBS module in the improved bottleneck layer includes Conv (convolution), BN (batch normalization) and SiLU (activation function), Upsample is an upsampling operation, and Concat is a feature concatenation operation. The first feature maps of different scales output by the improved backbone network (size is 14 2 ×48), the second feature map (size is 7 2 ×512) and the third feature map (size 12 ×1024) are respectively input into the second Concat module, the first Concat module, and the first CBS module of the improved bottleneck layer for processing, and the second C3CBAM module, the third C3CBAM module, and the fourth C3CBAM module of the improved bottleneck layer output the first spatial attention feature maps of different scales (size 14 2 ×256), the second spatial attention feature map (size 14 2 ×512), the third spatial attention feature map (size is 7 2 ×1024).

[0075] Furthermore, each C3CBAM module in the improved bottleneck layer is obtained by integrating the CBAM mechanism into the C3 module. The CBAM mechanism includes a channel attention module and a spatial attention module connected in sequence. Each C3CBAM module in the improved bottleneck layer performs global maximum pooling and global average pooling of the spatial dimension on the input feature map, uses the channel attention module to correct the input feature map, obtains the channel attention feature map, performs global maximum pooling and global average pooling of the channel dimension on the channel attention feature map, and uses the spatial attention module to correct the channel attention feature map to obtain the spatial attention feature map. The specific formula is as follows:

[0076]

[0077]

[0078] Where F represents the feature map input to each C3CBAM module, F∈R C×H×W , Mc represents the one-dimensional convolution of the channel attention module, M C ∈R C×1×1 , Ms represents the two-dimensional convolution of the spatial attention module, M S ∈R 1×H×W , F′ represents the channel attention feature map, and F″ represents the spatial attention feature map.

[0079] Specifically, global average pooling (getting average features) and global maximum pooling (getting the most significant features) help generate more accurate spatial attention maps. Figure 5By introducing the channel attention module, the receptive field is extended to the global feature map to enhance the representation ability of the target features: the input feature F is pooled, the weight T of each channel of the input feature F is calculated, and the feature G is obtained by integrating the weight T. The feature G is concatenated with the input feature F to obtain the channel attention feature map F′. By guiding the copper sleeve appearance defect detection model to give higher weights to the key areas of the target feature map, the discrimination of the target area can be improved; by introducing the spatial attention module, the spatial attention mechanism is used to encode wider spatial information into the local receptive field to enhance the expression ability of the feature map: through the channel attention feature map F′, the maximum pooling feature map R is obtained. q To highlight the most significant features of each position, we get the average pooling feature map R k Provide global average feature information to obtain the median pooling feature map R v Provides additional median information. Through the maximum pooling feature map R q And the average pooling feature map R k The similarity map matrix operation is used to generate the spatial attention weight matrix, which forms the spatial relationship between any two pixels in the mapping feature map, and then uses the median pooling feature map R v The median information provided is used to enhance the spatial relationship representation and is adaptively fused with the spatial attention weight matrix to obtain the spatial attention feature map F″.

[0080] Furthermore, the prediction layer in S1 includes three parallel Conv convolution layers, which are respectively connected one-to-one with the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module in the improved bottleneck layer. The three parallel Conv convolution layers receive and process the spatial attention feature maps output by the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module, and output predicted small, medium and large detection targets.

[0081] Specifically, see Figure 2 The prediction layer in the copper sleeve appearance defect detection model includes the first convolution layer (Conv1), the second convolution layer (Conv 2) and the third convolution layer (Conv 3) in parallel. The first convolution layer receives and processes the spatial attention feature map output by the fourth C3CBAM module and outputs a large detection target; the second convolution layer receives and processes the spatial attention feature map output by the third C3CBAM module and outputs a medium detection target; the third convolution layer receives and processes the spatial attention feature map output by the second C3CBAM module and outputs a small detection target.

[0082] S2. Collect image data containing copper sleeve appearance defects and create a copper sleeve appearance defect data set. The copper sleeve appearance defect data set is divided into a training set, a validation set and a test set. The training set is input into a copper sleeve appearance defect detection model for training. The trained model is verified and tested using the validation set and the test set. The model weights and parameters are updated in reverse order to obtain a copper sleeve appearance defect detection model after the test.

[0083] Furthermore, in S2, image data containing copper sleeve appearance defects are collected and a copper sleeve appearance defect data set is prepared, and the copper sleeve appearance defect data set is divided into a training set, a validation set and a test set. The specific process is as follows:

[0084] S21. Use an industrial camera to collect images of copper sleeve samples with appearance defects and normal copper sleeve samples, remove duplicate, irrelevant or poor quality data samples, and unify the image size and text encoding format to obtain processed images;

[0085] S22, performing image enhancement and balancing processing on the preprocessed image to obtain an expanded image; in order to expand the scale of the data set, the preprocessed image is enhanced by using data enhancement technology, and in order to avoid the problem of category imbalance, the preprocessed image is enhanced and balanced.

[0086] S23. Label the expanded image using the Labeling tool, and check the accuracy and consistency of the labeled image through cross-validation and review methods, thereby obtaining a copper sleeve appearance defect data set including several appearance categories; the several appearance categories are specifically: missing copper sheet, skewed stick, misaligned copper sheet, too long tail line, parallel copper sheet and qualified.

[0087] S24, dividing the copper sleeve appearance defect data set including several appearance categories into a training set, a verification set and a test set according to a preset ratio.

[0088] Specifically, in order to make up for the lack of copper sleeve appearance defect detection dataset in industrial scenarios, this application makes its own copper sleeve appearance defect dataset copper_sets. Figure 6 As shown, the copper sleeve model is PK1822. Figure 6 A to E in the figure represent the height of the copper sleeve, the outer diameter of the copper sleeve, the diameter of the stick, the height of the stick, and the inner diameter of the copper sleeve, which are 22 cm, 18 cm, 1 cm, 10 cm, and 10 cm respectively. The self-made copper sleeve appearance defect dataset contains 2400 images with image pixels of 1280×960 and a total of 6 categories of appearance (see Figure 7), 6 types of appearances include: Lack of copper (Lack_copper), Stick_defect, Upqralleled, Flying_wires, Corrected and True, 400 images for each type of appearance. According to the ratio of 8:1:1, they are divided into 1920 training sets, 240 validation sets and 240 test sets. The training set, validation set and test set are input into the copper sleeve appearance defect detection model for training, verification and testing, and then the model is analyzed, optimized and evaluated, and the model weights and parameters are updated in reverse to obtain the copper sleeve appearance defect detection model after testing.

[0089] S3. Acquire the RGB image of the copper sleeve to be detected from the copper sleeve production line in real time and perform data enhancement processing to obtain an image with increased data of the copper sleeve to be detected.

[0090] Specifically, the RGB image of the copper sleeve to be detected is subjected to mosaic data enhancement processing, and an image after data augmentation is obtained by adaptive anchor frame calculation and adaptive image scaling.

[0091] S4. Input the image after data enhancement of the copper sleeve to be detected into the copper sleeve appearance defect detection model after the test, output the detection target of the copper sleeve to be detected, and obtain the appearance defect detection result of the copper sleeve to be detected according to the detection target of the copper sleeve to be detected. The appearance defect detection result includes defect category and confidence.

[0092] Specifically, the image of the copper sleeve to be inspected after data enhancement is input into the tested copper sleeve appearance defect detection model for processing, and the predicted small, medium and large detection targets of the copper sleeve to be inspected are output: the small detection target detects small objects in the input image, which comes from the feature layer with higher resolution and can retain more details; the medium detection target detects medium-sized objects in the input image, and the resolution is between high resolution and low resolution; the large detection target detects large objects in the input image, which comes from the feature layer with lower resolution and captures large-area targets in the input image.

[0093] The prediction results of the three detection targets are aggregated and output on one picture to obtain the appearance defect detection results of the copper sleeve to be detected, that is, the detection boxes and categories of all targets.

[0094] In one embodiment, a copper sleeve appearance defect detection system is provided, which uses the above-mentioned copper sleeve appearance defect detection method based on the fusion attention mechanism for detection. The detection system includes: an image acquisition unit, an image processing unit, a copper sleeve appearance defect detection model and a computer system. The image acquisition unit is connected to the computer system, and the image processing unit is connected to the copper sleeve appearance defect detection model and is arranged in the computer system, wherein:

[0095] The image acquisition unit is used to collect image data of the copper sleeve to be inspected on the production line in real time, and send the collected image data to the computer system;

[0096] The computer system is used to receive the image data collected by the image collection unit and send it to the image processing unit;

[0097] The image processing unit is used to receive and process image data, and send the processed image data to the copper sleeve appearance defect detection model;

[0098] The copper sleeve appearance defect detection model is used to receive processed image data and perform feature extraction and prediction, and output the appearance defect detection results of the copper sleeve to be detected.

[0099] For the specific limitations of a copper sleeve appearance defect detection system, please refer to the above limitations of a copper sleeve appearance defect detection method based on a fusion attention mechanism, which will not be repeated here.

[0100] Furthermore, the effectiveness of a copper sleeve appearance defect detection method based on a fusion attention mechanism in the present invention is verified.

[0101] Ablation experiments were carried out on the copper sleeve surface defect detection model. Based on the YOLOv5 detection network, the original network structure was replaced with a lightweight network MobileNetV3S; then, the SE attention mechanism was incorporated into different backbone network layers, and qualitative and quantitative comparative experiments were carried out.

[0102] Based on the lightweight network MobileNetV3S, the SE (Squeeze-and-Excitation) attention mechanism is integrated into different network stages. Figure 8 , Fig. 9 As shown in Table 1, the first solution in this application adopts the YOLOv5 backbone network; the second solution adopts the MobileNetV3S backbone network; the third solution injects the SE attention mechanism into the second and third stages of the MobileNetV3S backbone network respectively, without changing the number of stages of the backbone network; the fourth solution adds the SE attention mechanism module between the second and third stages and between the third and fourth stages of the MobileNetV3S backbone network, which involves changes in the number of stages of the backbone network.

[0103] In order to intuitively evaluate the detection performance of the copper sleeve appearance defect detection model, training and testing were carried out on a self-made copper sleeve appearance defect dataset. Fig. 9As shown. In scheme 1, due to the complex network structure of YOLOv5 and its weak perception of local defect features, the average confidence obtained by YOLOv5 detection is lower than that of the lightweight network MobileNetV3S in scheme 2. Therefore, in schemes 3 and 4, the SE attention mechanism is introduced, both of which show higher detection confidence than the first two groups of experiments. This experiment proves that the attention mechanism can effectively enhance the perception of small defects in copper sleeves. Among them, the average confidence of the detection results of scheme 3 is higher than that of the other three groups. The confidence in the six defect scenarios is 98%, 100%, 89%, 93%, 83% and 87%, respectively, which are higher than scheme 4. Therefore, adding SE modules at different positions will affect the detection results. By comparing schemes 3 and 4, it is found that adding SE attention mechanism modules in the second and third stages of the MobileNetV3S backbone network can bring better detection performance.

[0104] In order to comprehensively evaluate the detection performance of different experimental groups, the data of four different target detection algorithms (corresponding to the four schemes in Table 1) under various indicators are compared, and the detection effect of each experimental group is qualitatively analyzed. The relevant experimental data comparison is shown in Table 1.

[0105] Table 1 Comparison of experimental data

[0106]

[0107] The experimental data in Table 1 show that the schemes 2, 3, and 4 using the MobileNetV3S backbone network are better than the scheme 1 using the basic network YOLOv5. The improved model shows shorter calculation time and better performance in detecting copper sleeve defects. Among them, scheme 3 performs better in various data indicators, with the precision, recall rate, average harmonic value of precision and recall rate and average precision values ​​of 0.977, 0.983, 0.979, and 0.987 respectively. It is particularly noteworthy that the average precision mAP@50 is improved by 0.7% compared with scheme 4. Due to the introduction of SE modules in the second and third stages of scheme 3, the model complexity increases by 8.07%, while this value is reduced by 25.33% compared with the original YOLOv5 network (corresponding to scheme 1). Experiments have shown that scheme 3, which integrates the SE attention mechanism into the lightweight MobileNetV3S network, can effectively enhance the sensitivity to local information features and focus on detail features, thereby improving the performance of copper sleeve detection. In summary, through qualitative and quantitative analysis of multiple groups of experiments and the introduction of Squeeze-and-Excitation modules at different locations of the backbone network, this paper proves that Scheme 3 achieves superior performance in detecting copper sleeve defects.

[0108] To further enhance the model's target detection capability, this paper introduces four attention mechanisms into the bottleneck layer and prediction layer. These four attention mechanisms include convolutional block attention module (CBAM), coordinate attention (CA), squeeze-excitation network (SE) and efficient channel attention (ECA), followed by qualitative and quantitative experimental analysis. In order to intuitively compare the detection performance of various attention mechanisms, this paper conducts performance tests on the self-made copper sleeve appearance defect dataset "copper_sets". The comparative effects of various attention mechanisms are shown in Figure 2. Fig.10 In order to better understand the focus of the detection network on the copper sleeve, the Grad-CAM tool was used to visualize and analyze six detection scenarios through heat maps, such as Fig.11 shown.

[0109] According to the detection results, it can be observed that in the case of a small number of exposed copper wires (Flying_Wires), the detection algorithm may miss the detection due to the small proportion of pixels representing exposed copper wires in the entire image. Only the two algorithms C3CA and C3CBAM with added attention mechanisms did not miss the detection in this case. Among them, the confidence scores of the C3CBAM algorithm are 89% and 69%, which are higher than C3CA. In addition, in various defect detections, the detection confidence of C3CBAM is also higher than that of other attention mechanisms, which is attributed to the introduction of spatial attention mechanism by C3CBAM. Therefore, adding CBAM helps the model focus on important information and key positions, effectively reducing missed detections and false alarms under different copper sleeve postures and external interference conditions.

[0110] In order to comprehensively evaluate the performance of the optimized C3 module and select the best performing module, a comparison of the performance of different attention mechanisms was carried out. The experimental results are shown in Table 2.

[0111] The experimental data in Table 2 show that C3CA, C3ECA and C3CBAM all show excellent performance. The integration of the attention mechanism significantly enhances the feature extraction capability of the C3 module and significantly improves indicators such as precision, recall rate, and refresh rate. Among them, C3CBAM performs best, with precision, recall rate, average precision and refresh rate reaching 97.8%, 98%, 98.3% and 13.09 respectively, which is the best performance among all attention mechanisms. Compared with the original network, the average precision is improved by 1.7%.

[0112] Table 2 Comparison results of multiple attention mechanisms

[0113] Model Name Accuracy Recall Average precision value Refresh rate Model complexity 1 C3 0.961 0.957 0.971 9.54 14.9 2 C3CA 0.974 0.967 0.978 13.33 14.9 3 C3ECA 0.969 0.959 0.967 13.12 14.9 4 C3CBAM 0.978 0.980 0.983 13.09 14.9

[0114] According to the experimental results, on the one hand, it is obvious that CA shows dependence on global information by setting channel weights through comprehensive analysis of feature map information. On the other hand, ECA solves the potential problem of over-reliance on global information in CA due to global average pooling operations by introducing a one-dimensional convolution kernel. Considering that global information may contain irrelevant content, which may degrade the performance of the model, ECA has an advantage over CA in tasks that emphasize local features, such as copper sleeve defect detection.

[0115] Compared with CA and ECA, CBAM integrates the advantages of convolution and attention mechanisms, and can fine-tune the attention control of image features from the dual perspectives of space and channel. By introducing the shared weight mechanism, the convolutional block attention module effectively establishes an inherent connection between channel and space, further enhancing the performance and generalization ability of the model. Therefore, in the case of the best average precision performance, the convolutional block attention module was selected as the attention mechanism.

[0116] The above-mentioned copper sleeve appearance defect detection method and system based on the fusion attention mechanism builds a copper sleeve appearance defect detection model, which includes an improved backbone network, an improved bottleneck layer and a prediction layer connected in sequence, the improved backbone network is used to extract features from the input image, the improved bottleneck layer is used to receive and process the feature map extracted by the backbone network, output the spatial attention feature map, the prediction layer is used to predict the spatial attention feature map, obtain the detection target corresponding to the input image, collect image data containing copper sleeve appearance defects and make a copper sleeve appearance defect data set, divide the copper sleeve appearance defect data set into a training set, a verification set and a test set, and train, verify and test the copper sleeve appearance defect detection model, reversely update the model weights and parameters, obtain the tested copper sleeve appearance defect detection model, obtain the RGB image of the copper sleeve to be detected from the copper sleeve production line in real time and perform data enhancement processing, input the enhanced image of the copper sleeve data to be detected into the tested copper sleeve appearance defect detection model for processing, and output the appearance defect detection result of the copper sleeve to be detected. This method can greatly improve the efficiency of copper sleeve appearance defect detection and improve the migration and generalization ability of the copper sleeve appearance defect detection model.

[0117] The above is a detailed introduction to a copper sleeve appearance defect detection method and system based on a fusion attention mechanism provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A copper sleeve appearance defect detection method based on fusion attention mechanism, characterized in that: The method comprises: S1. Based on the YOLOv5 target detection network, a copper sleeve appearance defect detection model is built. The model includes an improved backbone network, an improved bottleneck layer and a prediction layer connected in sequence. The improved backbone network is used to extract features from the input image, the improved bottleneck layer is used to receive and process the feature map extracted by the backbone network, and output the spatial attention feature map. The prediction layer is used to predict the spatial attention feature map to obtain the detection target corresponding to the input image. S2, collecting image data containing copper sleeve appearance defects and creating a copper sleeve appearance defect data set, dividing the copper sleeve appearance defect data set into a training set, a validation set and a test set, inputting the training set into a copper sleeve appearance defect detection model for training, and using the validation set and the test set to verify and test the trained model, and reversely updating the model weights and parameters to obtain a copper sleeve appearance defect detection model after the test; S3, acquiring the RGB image of the copper sleeve to be detected from the copper sleeve production line in real time and performing data enhancement processing to obtain an image with increased data of the copper sleeve to be detected; S4. Input the image after data enhancement of the copper sleeve to be detected into the copper sleeve appearance defect detection model after the test, output the detection target of the copper sleeve to be detected, and obtain the appearance defect detection result of the copper sleeve to be detected according to the detection target of the copper sleeve to be detected. The appearance defect detection result includes defect category and confidence.

2. The copper sleeve appearance defect detection method based on the fusion attention mechanism according to claim 1 is characterized in that: The improved backbone network in S1 includes a Conv_BN module, a first bneck module introducing the SE attention mechanism, a second bneck module introducing the SE attention mechanism, and a third bneck module to an eleventh bneck module connected in sequence. The image to be detected is input into the Conv_BN module for processing, and the feature map output by the Conv_BN module is processed and features are extracted respectively by the first bneck module introducing the SE attention mechanism, the second bneck module introducing the SE attention mechanism, and the third bneck module to the eleventh bneck module connected in sequence, and feature maps of different scales are output through the seventh bneck module, the tenth bneck module and the twelfth bneck module.

3. The copper sleeve appearance defect detection method based on the fusion attention mechanism as claimed in claim 2 is characterized in that: Each backneck module in the improved backbone network in S1 that introduces the SE attention mechanism extracts features from the input feature map. The specific process is as follows: S11. Each backneck module that introduces the SE attention mechanism receives the feature map input by the previous module adjacent to it, and generates convolution features through convolution operation; S12, performing a global average pooling operation on the convolutional features, compressing the two-dimensional features of each channel, and obtaining a feature vector; S13, processing the feature vector through two fully connected layers connected in sequence, normalizing the processed feature vector through an activation function, and obtaining a channel weight value; S14. Weight the first feature map through the channel weight value to obtain the feature map output by each backneck module that introduces the SE attention mechanism.

4. The copper sleeve appearance defect detection method based on the fusion attention mechanism as claimed in claim 3 is characterized in that: Each bneck module in the improved backbone network in S1 includes several bneck blocks, each of which includes a deep packet convolution layer, a pooling layer, and a fully connected activation function layer connected in sequence. Each bneck block uses a deep packet convolution method to extract features from the input feature map, which specifically includes the following steps: S15, the deep grouped convolution layer of each bneck block receives the feature map output by the previous module adjacent to it, reduces the number of channels of the feature map by half, obtains the first feature and the second feature with half the number of channels, performs deep grouped convolution processing on the first feature with half the number of channels, obtains the third feature, and fuses and optimizes the third feature with the second feature with half the number of channels to form a dense feature map containing more appearance information; S16, dividing the dense feature map into several rectangular regions and performing pooling processing through a pooling layer, reducing the dimension and width of each rectangular region to prevent overfitting, and obtaining a maximum region feature map; S17, the fully connected activation function layer receives the maximum area feature map and converts it into a one-dimensional feature vector to obtain a feature map containing the target global features output by each bneck block.

5. The copper sleeve appearance defect detection method based on the fusion attention mechanism as claimed in claim 4 is characterized in that: The improved bottleneck layer in S1 includes a first CBS module, a first Upsample module, a first Concat module, a first C3CBAM module, a second CBS module, a second Upsample module, a second Concat module, a second C3CBAM module, a third CBS module, a third Concat module, a third C3CBAM module, a fourth CBS module, a fourth Concat module, and a fourth C3CBAM module connected in sequence. The first CBS module is also connected to the fourth Concat module, and the second CBS module is also connected to the third Concat module. The first CBS module, the first Concat module, and the third Concat module in the improved bottleneck layer respectively receive and process feature maps of different scales output by the improved backbone network, and output spatial attention feature maps of different scales through the second C3CBAM module, the third C3CBAM module, and the fourth C3CBAM module.

6. The copper sleeve appearance defect detection method based on the fusion attention mechanism as claimed in claim 5 is characterized in that: Each C3CBAM module in the improved bottleneck layer is obtained by integrating the CBAM mechanism into the C3 module. The CBAM mechanism includes a channel attention module and a spatial attention module connected in sequence. Each C3CBAM module in the improved bottleneck layer performs global maximum pooling and global average pooling of the spatial dimension on the input feature map, uses the channel attention module to correct the input feature map, obtains the channel attention feature map, performs global maximum pooling and global average pooling of the channel dimension on the channel attention feature map, and uses the spatial attention module to correct the channel attention feature map to obtain the spatial attention feature map. The specific formula is as follows: Where F represents the feature map of each C3CBAM module input, Mc represents the one-dimensional convolution of the channel attention module, Ms represents the two-dimensional convolution of the spatial attention module, F′ represents the channel attention feature map, and F″ represents the spatial attention feature map.

7. The copper sleeve appearance defect detection method based on fusion attention mechanism as claimed in claim 6 is characterized in that: The prediction layer in S1 includes three parallel Conv convolution layers, which are connected one-to-one with the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module in the improved bottleneck layer respectively. The three parallel Conv convolution layers receive and process the spatial attention feature maps output by the second C3CBAM module, the third C3CBAM module and the fourth C3CBAM module respectively, and output the predicted small, medium and large detection targets.

8. The copper sleeve appearance defect detection method based on the fusion attention mechanism according to claim 7 is characterized in that: In S2, image data containing copper sleeve appearance defects are collected and a copper sleeve appearance defect dataset is prepared. The copper sleeve appearance defect dataset is divided into a training set, a validation set, and a test set. The specific process is as follows: S21, using an industrial camera to collect and process images of the copper sleeve sample with appearance defects and the normal copper sleeve sample to obtain processed images; S22, performing image enhancement processing and balancing processing on the pre-processed image to obtain an expanded image; S23, using a Labeling tool to label the expanded image, and checking the accuracy and consistency of the labeled image, thereby obtaining a copper sleeve appearance defect data set including several types of appearances; S24, dividing the copper sleeve appearance defect data set including several types of appearance into a training set, a verification set and a test set according to a preset ratio.

9. The copper sleeve appearance defect detection method based on the fusion attention mechanism as claimed in claim 8 is characterized in that: The several categories of appearance are: missing copper pieces, crooked sticks, misaligned copper pieces, too long tail wires, parallel copper pieces, and qualified.

10. A copper sleeve appearance defect detection system, using the copper sleeve appearance defect detection method based on fusion attention mechanism as described in any one of claims 1 to 9 for detection, characterized in that: The detection system comprises: an image acquisition unit, an image processing unit, a copper sleeve appearance defect detection model and a computer system, the image acquisition unit is connected to the computer system, the image processing unit is connected to the copper sleeve appearance defect detection model and is arranged in the computer system, wherein: The image acquisition unit is used to collect image data of the copper sleeve to be inspected on the production line in real time, and send the collected image data to the computer system; The computer system is used to receive the image data collected by the image collection unit and send it to the image processing unit; The image processing unit is used to receive and process image data, and send the processed image data to the copper sleeve appearance defect detection model; The copper sleeve appearance defect detection model is used to receive processed image data and perform feature extraction and prediction, and output the appearance defect detection results of the copper sleeve to be detected.

Images