Aluminum profile defect detection and identification method based on YOLOv7-ESC

Abstract

The invention relates to a YOLOv7-ESC-based aluminum profile defect detection and identification method, which comprises the following steps of S1, obtaining a public data set, and carrying out data preprocessing by adopting a Tianpool aluminum profile surface defect identification data set; s2, a YOLOv7 model is improved, and 3 * 3 Conv convolution in a backbone network ELAN module is replaced with PCnov convolution; s3, constructing a residual pyramid pooling structure, and inputting the features extracted by the backbone network into a residual pyramid pooling module SPPCSPC-F; s4, adding a CBAM attention mechanism between the backbone network and the feature fusion network; s5, replacing the regression loss function CIoU with a WIoU loss function, and transmitting the features after feature fusion into a detection head part for classification prediction; s6, the YOLO network model is trained, and the optimal weight is reserved; and S7, testing by utilizing the trained optimal weight, evaluating a detection result, and finally realizing automatic and intelligent identification of the surface defects of the aluminum profile. According to the method, the aluminum profile surface defect detection precision can be improved, and the small target defect omission ratio is reduced.

Description

Technical Field

[0001] The present invention relates to the field of aluminum profile surface defect detection, and in particular to an aluminum profile defect detection and recognition method based on YOLOv7-ESC. Background Art

[0002] Aluminum profiles are the most widely used non-ferrous metal materials in industrial applications. They are widely used in aviation, aerospace, automobiles, machinery manufacturing, construction and other fields. They are favored for their lightweight, corrosion resistance and good plasticity. In the process of producing aluminum profiles in factories, they need to go through a variety of process flows. Each step of the processing technology plays a key role in the quality and performance of aluminum profiles, but surface defects are inevitable in this process. Surface defects of aluminum profiles will affect the appearance, quality and performance of aluminum profiles, and seriously lead to the inability to sell them normally. It is very important to detect aluminum profile defects quickly and accurately. Defect detection is required before leaving the factory to avoid adverse effects on the aesthetics and performance of the product. Traditional manual detection methods rely on the subjective judgment of operators, and there are problems such as misjudgment, missed detection, low accuracy and efficiency. Later, the emergence of image processing algorithms based on machine vision improved these problems to a certain extent, but the application of machine vision to surface defects of aluminum profiles with a large number of defects has the problems of complex feature extraction and poor generalization, and the effect of small target defect detection is not ideal.

[0003] With the improvement of computer performance and the advent of the big data era, deep learning algorithms have become a hot topic, and have made breakthrough progress in natural language processing, image classification, target detection and data mining. At the same time, they have also made up for the shortcomings of traditional detection methods in aluminum profile surface defect detection. Deep learning algorithms automatically extract high-level features of input images through neural networks, and perform classification, detection and other operations, eliminating the tedious manual feature extraction process and improving detection efficiency.

[0004] At present, the aluminum profile defect detection methods are mainly divided into two-stage detection algorithms represented by Faster R-CNN and single-stage detection algorithms represented by the YOLO series. The two-stage detection method has a deeper network structure and higher detection accuracy, but the detection speed is slow, and it is difficult to meet user needs in real-time target detection. The single-stage detection method, because of its simpler network structure, can improve the detection speed while ensuring accuracy, and can achieve real-time detection. However, due to the existence of defects such as dirty spots and scratches in aluminum profiles, compared with other types of defects, the pixel area they occupy in the image is too small and the features are not obvious, resulting in poor defect detection effect. The YOLO model still needs to improve the detection accuracy of such small target surface defects. There are also some other problems in the aluminum profile defect detection algorithm, such as large model size, detection speed needs to be improved, insufficient feature extraction capability, resulting in missed detection of weak long strip defects and missed detection of small defects. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an aluminum profile defect detection and recognition method based on YOLOv7-ESC, which can improve the detection accuracy and detection speed, reduce the missed detection of small target defects, and reduce the number of model parameters; it has good robustness and generalization ability.

[0006] The technical solution adopted by the present invention is:

[0007] The aluminum profile defect detection and recognition method based on YOLOv7-ESC includes the following steps:

[0008] Step 1: First, obtain the public aluminum profile surface defect dataset from Alibaba Cloud Tianchi, then perform data preprocessing on the aluminum profile surface images and labels to achieve the effects of denoising and image enhancement, and obtain an aluminum profile surface defect map with prominent defects. Finally, establish the YOLOv7-ESC network framework;

[0009] Step 2: Improve the ELAN module, replace the Conv convolution module in the backbone network ELAN module with the PCnov convolution module to extract the surface defects of aluminum profiles; compared with the ordinary convolution module, the PConv convolution module is characterized by using ordinary convolution to extract part of the features of the input channel and output channel for calculation, which can reduce redundant calculations and memory access without losing the number of channel features, and more effectively extract defect features. The network calculation amount, parameter amount and complexity are reduced, and the model is more lightweight;

[0010] Step 3: Reconstruct the SPPCSPC module and construct the residual pyramid pooling structure SPPCSPC-F. The features extracted by the backbone network are input into the residual pyramid pooling structure SPPCSPC-F.

[0011] Step 4: Add the CBAM attention mechanism module between the backbone network and the feature fusion network;

[0012] Step 5: The regression loss function CIoU is replaced by the WIoU loss function, and the features fused by the feature fusion network are passed to the detection head for classification prediction;

[0013] Step 6: Train the YOLOv7-ESC network model. First, divide the aluminum profile surface defect dataset into training set, validation set and test set in a ratio of 4:1:1. Use Mosaic to further enhance the data during the training process. Then input it into the YOLOv7-ESC network model for pre-training, and take the pre-training weight with the smallest training loss value for transfer learning. Finally, combine the aluminum profile surface defect dataset and the pre-training weight for training to obtain the optimal weight;

[0014] Step 7: Use the trained optimal weights to conduct tests and evaluate the test results, ultimately achieving automated and intelligent identification of aluminum profile surface defects.

[0015] In step 1, the produced aluminum profile images are first collected and sorted to obtain a data set of aluminum profile surface defects, including dents, non-conductivity, scratches, orange peel, bottom leakage, bruises, pits, convex powder, coating cracks, and dirty spots, with a total of 10 categories and 1,885 color images.

[0016] In the step 1, the aluminum profile surface image is subjected to data enhancement and data preprocessing, the defect area is annotated using Labelme software, and a json file is generated to save the annotation information; then the aluminum profile surface defect image is subjected to data preprocessing to achieve the effects of denoising and image enhancement, and a surface defect image of the aluminum profile with prominent defects is obtained. Specifically, it includes:

[0017] Traditional image processing is performed on the aluminum profile surface image. First, the enhanced aluminum profile surface defect image is converted into RGB color space and HSI color space, and the components are extracted as required for grayscale transformation. Then, the image noise is analyzed and the local median filtering algorithm is proposed and applied to process the noise, and the surface defect map of the aluminum profile with prominent defects is obtained.

[0018] In step 1, the YOLOv7-ESC network framework structure includes: a backbone network, a neck feature fusion network, and a detection head. Figure 4 As shown in the figure. The backbone network extracts the surface defect features of aluminum profiles in the input image. First, the input feature image is 640×640×3, and the features are refined and extracted through four layers of CBL convolution blocks. Then, the ELAN-P module reduces the number of parameters and downsamples to reduce the number of feature channels. Next, the feature map of size 20×20×1024 is input to the neck feature fusion network. The neck feature fusion network uses top-down and bottom-up connections to perform feature fusion processing on multi-scale feature layers. Finally, it is sent to the detection head to output feature maps of different scales for loss calculation, defect location and defect prediction.

[0019] In step 2, the 3*3Conv convolution module in the ELAN module is replaced by a PConv convolution module, which can reduce the number of module parameters. The ELAN-P module consists of a short path and a long path, the short path is a 1*1CBS convolution, and the long path is composed of a 1*1CBS convolution and 4 consecutive PConv convolutions. The PConv convolution module is characterized in that it uses ordinary convolution to extract part of the features of the input channel and the output channel for calculation, which can reduce redundant calculations without losing the number of channel features and speed up the model reasoning speed.

[0020] The specific principles are as follows:

[0021] Assume that the size of the input feature map X is c×h×w, where h, w, and c represent the height, width, and number of channels of the input feature map X, respectively. Input feature feature I∈R c×h×w , the output is O∈R c×h×w , the standard convolution kernel size is w∈R c×k×k , k represents the convolution kernel size. Since the PConv convolution module only needs to feature extract part of the channel cp, the convolution kernel size is reduced to w∈R cp×k×k The formula for the ratio of PConv convolution to Conv convolution calculation is as follows:

[0022]

[0023] In the above formula, FLOPS PConv Indicates PConv calculation amount; FLOPS Conv represents the amount of PConv calculation; h represents the height of the input feature map; w represents the width of the input feature map; c p represents the number of channels extracted by PConv; c represents the number of channels of the input feature map.

[0024] In step 2, in the PConv convolution module, in order to avoid the high redundancy caused by different channels of the feature map, the convolution operation is performed on only part of the input channels, while the remaining input channels remain unchanged. The cp channels of the input are used as representatives of the entire feature map for calculation. At this time, the calculation amount of PConv is h×w×k 2 ×c p 2 If cp is set to 1 / 4 of the input channel c, the amount of computation is 1 / 16 of that of regular convolution, and the memory access is 1 / 4 of that of regular convolution.

[0025] In step 3, a residual pyramid pooling structure SPPCSPC-F is constructed, and the features extracted by the backbone network are input into the constructed residual pyramid pooling structure. The residual pyramid pooling structure SPPCSPC-F includes an upper multi-scale feature extraction channel and a lower small target feature extraction channel. Its structure diagram is shown in FIG. Figure 2 As shown in the figure, the upper multi-scale feature extraction channel captures feature information of different scales through different pooling kernels to obtain richer feature target information, and a 1×1 convolution adjustment channel is used before and after the pooling layer; the lower small target feature extraction channel introduces three layers of residual convolution blocks, and three consecutive linear residual convolution blocks CF modules refine the extracted features and extract more small target features.

[0026] The CF module consists of convolution and activation function FReLU. The activation function FReLU can improve the spatial sensitivity of multi-scale defect information of aluminum profiles. The tensor addition is realized through the add operation, the number of feature channels remains unchanged to increase the amount of information, and the 1×1 convolution feature is fused.

[0027] In step 3, the features extracted by the backbone network are input into the residual pyramid pooling structure SPPCSPC-F. First, the input features are divided into two parts, one of which is subjected to residual convolution processing to refine the extracted features. The other part is processed by the SPP structure, and four layers of pooling layers of different sizes output four layers of different receptive fields to facilitate the distinction between large and small targets. Finally, the two parts are merged together. It enhances the multi-scale feature learning ability of the network, captures more small target defect features, and solves the problem of missed detection of small target defects in aluminum profile surface defects.

[0028] In the step 4, a CBAM attention mechanism module is added after the residual pyramid pooling structure SPPCSPC-F, and a channel attention mechanism and a spatial attention mechanism are introduced into the network to improve the perception ability of the model. Without significantly increasing the model parameters and scale, the model performance is quickly expanded and the features are adaptively refined, the loss of position information is reduced, more attention is paid to small target information, and the accuracy of small target detection is increased.

[0029] The step 4 comprises the following steps:

[0030] S4.1. Input the feature map in the channel dimension, pass through the parallel MaxPool layer and AvgPool layer, get two 1×1 channel weight matrices, change the feature map dimension from c×h×w to c×1×1, compress the number of channels to 1 / r times the original through the MLP module, expand to the original number of channels through the ReLU activation function, add it to the original feature map, get the channel attention output result through the sigmoid activation function, and multiply it with the original image to generate the intermediate feature map (c represents the number of feature map channels, r represents the parameter for adjusting the number of channels). The weights W0 and W1 of the MLP module are shared for the input. The channel attention mechanism is expressed as follows:

[0031] M c (F)=σ(MLP(AvgPool(Fm))+MLP(MaxPool(Fm))

[0032] Where M c (F) represents the channel attention mechanism; σ represents the sigmoid activation function; W0 and W1 represent weight coefficients, W0∈R c / r×c , W1∈R c×c / r ; AvgPool(Fm) represents the feature map processed by the average pooling layer; MaxPool(Fm) represents the feature map processed by the maximum pooling layer; MLP(AvgPool(Fm)) represents the feature map processed by the average pooling layer and the MLP module; MLP(MaxPool(Fm)) represents the feature map processed by the maximum pooling layer and the MLP module.

[0033] S4.2. Input the feature map in the spatial dimension, apply average pooling and maximum pooling operations to the feature map F output by the channel attention module, and the new feature map is denoted as F S avg and F S max , F S avg represents the feature map of channel dimension average pooling, F S max It represents the feature map with the maximum pooling of the channel dimension, with a size of 1×h×w; then the new feature maps obtained by the two pooling operations are connected. Finally, the convolution operation feature map is applied to generate a spatial attention feature map of size R×h×w.

[0034] The spatial attention mechanism is expressed as follows:

[0035] M s (F) = σ(f 7×7 ([AvgPool(Fm);MaxPool(Fm)]))

[0036] Where M s (F) represents the spatial attention mechanism, f 7×7 Represents the convolution process with a convolution kernel size of 7×7.

[0037] In step 5, the regression loss function CIoU is replaced by the WIoU loss function, and the features fused by the feature fusion network are passed to the detection head for classification prediction.

[0038] WIoU is a bounding box loss function that evaluates the quality of the aiming box. It has a dynamic focusing function and can better measure the similarity between the predicted result of the detection box and the real annotation box, which can improve the performance of target detection. The WIoU loss function formula is as follows:

[0039]

[0040] Where α is a hyperparameter; β is the outlier degree; θ represents the Euclidean distance between the center point of the predicted box boundary and the center point of the true box boundary; r represents the gradient gain; w c 、h c L are the width and height of the minimum bounding rectangle of the predicted box and the real box respectively; WIOU represents the WIoU loss, L IOU represents the intersection-over-union loss. x represents the horizontal coordinate of the center point of the prediction box, y represents the vertical coordinate of the center point of the prediction box, and x represents the horizontal coordinate of the center point of the prediction box. t Indicates the horizontal coordinate of the center point of the target frame, y t Indicates the vertical coordinate of the center point of the target box.

[0041] The training process inevitably includes low-quality samples. Factors such as aspect ratio and distance will further increase the penalty for low-quality samples, causing the model to reduce its generalization ability. When the target box and anchor box overlap more, the penalty for geometric factors should be reduced. Reducing intervention in training can enhance the generalization ability of the model. The dynamic non-monotonic focused gradient gain allocation strategy is adopted to reduce the competitive advantage of high-quality anchor boxes and mitigate the adverse gradient effects generated by low-quality samples, so that the model has higher accuracy. Focus more on high-quality anchor boxes in the initial stage of training, assign smaller weights to low-quality anchor boxes, and reduce harmful gradients.

[0042] The step 6 comprises the following steps:

[0043] S6.1. Randomly divide the aluminum profile surface defect dataset into training set, validation set and test set in a ratio of 4:1:1; randomly generate a reference point in the mosaic canvas with coordinates (x, y), where represents the horizontal coordinate of the reference point and represents the vertical coordinate of the reference point. The reference point is randomly generated. The coordinates of the vertex in the upper right corner of the mosaic canvas are (0, 0);

[0044] S6.2, as shown in FIG6(a), according to the generated mosaic canvas, the original image is placed at the upper left, upper right, lower left, and lower right positions of the mosaic canvas reference point according to a fixed size; the size of the mosaic canvas is generally fixed, which is twice the size of the original image, and the canvas is the image after mosaic data enhancement;

[0045] S6.3, according to the size transformation method of each image, correspond the mapping relationship to the image label;

[0046] S6.4, as shown in Figure 6(b), the mosaic canvas is spliced ​​according to the randomly generated horizontal and vertical coordinates of the reference points, and four original images are randomly selected for edge processing in the mosaic canvas. For example, when the upper right corner of the original image exceeds the upper right corner edge of the canvas, it is necessary to process the part of the image that exceeds the canvas and retain the image area within the canvas;

[0047] S6.5. Input the sorted data set into the YOLOv7-ESC network model for pre-training, and take the pre-training weight with the smallest training loss value for transfer learning. Combine the aluminum profile surface defect data set and the pre-training weight for training to obtain the optimal weight.

[0048] In step 7, the detection results are qualitatively evaluated, and the evaluation indicators include average precision mean, recall rate, model parameter quantity, calculation quantity, parameter quantity and number of frames.

[0049] The present invention provides an aluminum profile defect detection and recognition method based on YOLOv7-ESC, and the technical effects are as follows:

[0050] 1) In step 1 of the present invention, the data preprocessing of the aluminum profile surface image and label is an indispensable part of deep learning, which can improve the quality of data and the performance of the model. By preprocessing the training data, the accuracy of the model prediction can be enhanced.

[0051] Labeling is an important part of image preprocessing. In the image dataset of aluminum profiles, accurate labels provide clear learning goals for the model, helping the model to identify and classify features in the image and improve the accuracy of prediction. Label the defect area through the image annotation tool Labelme software to ensure the consistency and accuracy of the label, reduce the deviation and error in the data, improve the generalization ability of the model, reduce the risk of overfitting, and thus improve the learning effect of the model. Then, the collected aluminum profile surface defect images are converted between RGB color space and HSI color space. The conversion between RGB and HSI color space not only helps to improve the accuracy and robustness of defect detection, but also simplifies the image processing process, providing stronger support for the analysis of surface defects.

[0052] 2) In step 2 of the present invention, in the industrial context, the aluminum profile surface defect detection model needs to focus on practicality and often needs to be deployed in embedded devices for detection, so the detection model needs to meet the accuracy requirements while being lightweight. The partial convolution PConv convolution module is used in the ELAN-P module to reduce the amount of calculation, parameter quantity and complexity of the entire model, speed up the inference speed, and make the model more lightweight, which is conducive to the deployment of the model on embedded devices.

[0053] Compared with the ordinary convolution Cnov module, the PConv convolution module is characterized by using ordinary convolution to extract part of the features of the input channel and the output channel for calculation, which can reduce redundant calculations and memory access without losing the number of channel features, and more effectively extract defect features. PConv has a lower computational complexity than conventional convolution. In addition, the PConv convolution module better utilizes the capabilities of computing devices and shows good results in extracting spatial features at the same time. Therefore, the use of the PConv convolution module can not only keep the model lightweight, but also enhance the ability to generate feature maps, which brings significant advantages to the training and reasoning of aluminum profile surface defect detection models.

[0054] 3) In step 3 of the present invention, there are certain similarities between different defects in aluminum profile defect detection, the background information is complex, and the defect scales vary greatly. The SPPCSPC module is used to realize the fusion of feature maps of different scales, build associations between targets of different scales, and capture features of different scales. However, when processing small targets, it may still be unable to effectively extract sufficient context information, resulting in a decrease in overall detection performance.

[0055] Therefore, the residual pyramid pooling structure SPPCSPC-F is constructed to enhance the module's attention to small target defect information, better capture multi-scale defect information, and improve feature extraction capabilities. The residual structure in the residual pyramid pooling structure SPPCSPC-F enhanced module can better propagate gradients during training, improve model performance, learn multi-scale features, and provide module feature extraction capabilities.

[0056] 4) In step 4 of the present invention, the background information in aluminum profile detection is complex. In this case, it is difficult to detect aluminum profile surface defects. In order to improve the model's ability to detect aluminum profile surface defects and suppress redundant information, a CBAM attention mechanism module is added between the backbone network and the feature fusion network.

[0057] The CBAM attention mechanism module is an attention module for feedforward convolutional neural networks. It can combine cross-channel information and spatial information to extract information features, allowing the model to adaptively learn the importance of channels and spatial positions, helping the model to focus on important features and suppress unimportant features, improving the model's detection capabilities without incurring excessive computational effort and parameter volume.

[0058] 5) In step 5 of the present invention, the loss function is used to measure the difference between the predicted label and the true label. A good loss function should weaken the penalty of geometric factors when the target box and the true box overlap, so that the model can obtain better generalization ability. A suitable loss function can accelerate the convergence of the model, thereby improving the accuracy and stability of target detection.

[0059] The original coordinate loss function CloU of YOLOv7 uses a complex function calculation to not only consider the overlap between the target frame and the real frame and the distance between the center points, but also measures the aspect ratio. It can more accurately judge the similarity between the predicted frame and the real frame, but it consumes a lot of computing power during the large number of calculations, which increases the training time. When the loss function CloU measures the small target defect detection frame in the surface defects of aluminum profiles, it is very sensitive to the position deviation of small targets and has poor small target recognition ability. Therefore, the dynamic non-monotonic focusing mechanism, namely the WIoU loss function, is used to replace the original CIoU loss function of YOLOv7.

[0060] 6) In step 6 of the present invention, Mosaic technology generates a new synthetic image by stitching multiple images together. This method can effectively increase the diversity of the data set, help the model learn richer features, and thus improve the generalization ability of the model. Through Mosaic enhancement, the generated synthetic image can provide more information in one training, thereby reducing the number of training rounds required, shortening the training time, and improving the training efficiency.

[0061] Pre-training enables the model to learn rich features on the aluminum profile surface defect dataset, which can significantly reduce the training time when applied to subsequent training tasks, because the initial weights of the network are already relatively reasonable, avoiding the time consumption of training from scratch and improving the model learning efficiency. At the same time, model pre-training can effectively reduce the risk of overfitting when training on small datasets, improve the accuracy of the model, and enhance the robustness of the model.

[0062] 4) Compared with the original YOLOv7 model, the YOLOv7-ESC network model proposed in this invention reduces the model volume, parameter quantity and calculation amount by 16.7%, 16.9% and 23.4% respectively, improves the model detection accuracy, and reduces the missed detection rate. The FPS is increased by 5.01, and the increased detection speed can bring better detection efficiency.

[0063] 5) The present invention proposes a YOLOv7-ESC network model for detecting 10 surface defects of aluminum profiles, which can meet the needs of real-time performance in industrial detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0065] Figure 1 This is a flow chart of the aluminum profile surface defect detection and recognition method based on YOLOv7-ESC;

[0066] Figure 2 This is the SPPCSPC-F module structure diagram;

[0067] Figure 3 This is the ELAN-P module structure diagram;

[0068] Figure 4 This is the structure diagram of the aluminum profile surface defect detection model of YOLOv7-ESC;

[0069] Figure 5 This is a neural network recognition effect diagram obtained in the embodiment;

[0070] Figure 6(a) is a schematic diagram of the Mosaic data enhancement principle;

[0071] FIG6( b ) is a schematic diagram of stitching the mosaic canvas. DETAILED DESCRIPTION

[0072] like Figure 1 As shown in the figure, the aluminum profile surface defect detection and recognition process based on YOLOv7-ESC includes the following steps:

[0073] S1. Obtain a public data set, then preprocess the aluminum profile surface images and labels, and establish the YOLOv7-ESC network framework.

[0074] Firstly, the bare aluminum profiles are collected and sorted to obtain a data set of aluminum profile surface defects. The defects include dents, non-conductivity, scratches, orange peel, leakage, bruises, pits, convex powder, coating cracks, and dirty spots, with a total of 1885 color pictures in 10 categories. Then, traditional image processing is performed on the aluminum profile surface images to make it easier for the YOLOv7-ESC network to extract defect information.

[0075] Traditional image processing is performed on the surface image of aluminum profiles. First, the collected surface defect images of aluminum profiles are converted into RGB color space and HSI color space, and components are extracted as required for grayscale transformation. Then, the noise of the image is analyzed and a local median filtering algorithm is proposed and applied to process the noise, thus obtaining a surface defect map of aluminum profiles with prominent defects.

[0076] RGB color space:

[0077] The principle of RGB color space is to use Cartesian coordinates to build a cube. The three primary colors R, G, and B correspond to three coordinate axes respectively, and gray levels are assigned along each coordinate axis. In this coordinate model, the origin represents black, and the vertices on the cube on the coordinate plane correspond to cyan, crimson, and yellow respectively. White is the point farthest from black, that is, the vertex farthest from the origin. All colors are normalized, that is, the values ​​of R, G, and B are taken on [0,1]. Various colors can be defined using vectors distributed at the origin. It can be seen that different colors are located inside or on the surface of the unit cube.

[0078] HSI color space:

[0079] In order to segment the aluminum profile surface plate image based on color and separate the feature area of ​​interest on a separate plane, a component of the HSI color space is used as a template image. The brightness represented by the I component has nothing to do with the color information of the aluminum profile surface plate, and the hue represented by the H component and the saturation represented by the S component have a mathematical model with the way human vision perceives color. Therefore, the HIS color space can be used as the color feature of the human visual system to study image processing algorithms.

[0080] Grayscale transformation:

[0081] Grayscale transformation method is an important method of spatial domain image enhancement. It can effectively improve the quality of aluminum profile surface images, enhance the contrast of aluminum profile surface images, highlight more details of the target area and the inherent characteristics of aluminum profile surface images, and also remove unnecessary features from the image.

[0082] Median filtering can be expressed as follows:

[0083] g(x,y)=med{f(xw,yh),(w,h∈S)}

[0084] In the above formula, S is a two-dimensional template, i.e., the convolution kernel, with commonly used sizes of 3×3 and 5×5; f(xw,yh) represents the pixel grayscale value of the template window; f(x,y) and g(x,y) are the original image and the processed image, respectively.

[0085] like Figure 4 As shown in the figure, the YOLOv7-ESC network framework structure includes: a backbone network, a neck fusion network and a detection head. The backbone network extracts the surface features of the aluminum profile in the image. The neck fusion network uses a top-down and bottom-up connection method to fuse the multi-scale feature layers, and finally sends them to the detection head for loss calculation and defect prediction.

[0086] S2. Replace the 3*3Conv module in the network ELAN module with the PCnov module to lightweight the network and reduce redundant information extraction.

[0087] The feature extraction module ELAN-P is as follows Figure 3 As shown in the figure, it contains 4 consecutive PConv feature extractions. Compared with the Conv module, the Pconv module uses Conv convolution to extract part of the features of the input channel and the output channel for calculation. It can reduce redundant calculations and memory access without losing the number of channel features, and more effectively extract defect features. The network calculation amount, parameter amount and complexity are reduced, and the model is more lightweight.

[0088] S3. Surface defects of aluminum profiles include dents, non-conductivity, scratches, orange peel, leaking bottom, bruises, pits, convex powder, coating cracks, and dirty spots. The scales of aluminum profile defects vary greatly, and the detection accuracy of small target defects is not high and it is easy to miss. Construct the residual pyramid pooling SPPCSPC-F module, and the feature map after the backbone network is downsampled is passed to the SPPCSPC-F module for feature extraction to extract more multi-scale defect information.

[0089] The spatial pyramid pooling SPPCSPC-F module is as follows Figure 2 As shown. First, the input features enter two channels, one of which is subjected to continuous residual convolution processing. The add operation does not change the number of channels to obtain more defect information, while refining the extracted features. The other channel is processed by the SPP structure, and four layers of pooling layers of different sizes output four layers of different receptive fields, which is convenient for distinguishing large targets from small targets. Finally, the two channels are merged together through the concat full connection operation. The reconstructed pyramid pooling module enhances the network's multi-scale feature learning ability, captures more defect information, improves the model's attention to small target information, and solves the problem of missed detection of small target defects on the surface of aluminum profiles.

[0090] S4. Add the CBAM attention mechanism module to the neck network. FPN is a bottom-up multi-scale fusion network that fuses high-level features to low-level features, increases the expressiveness of low-level features, and improves performance. Targets of different scales can be assigned to different layers for prediction, achieving divide and conquer. CBAM is channel space attention, which obtains the importance of each feature channel and feature space through automatic learning, and then uses this importance to assign a weight value to each feature channel and space, so that the neural network can focus on it, thereby solving the problem of different scale fusion leading to the loss of small target defect features on the surface of aluminum profiles. The CBAM module combines the channel attention mechanism and the spatial attention mechanism into the network, and quickly expands the model performance without significantly increasing the model parameters and scale.

[0091] The specific steps include:

[0092] S4.1. Input the feature map in the channel dimension, pass through the parallel MaxPool layer and AvgPool layer, get two 1×1 channel weight matrices, change the feature map dimension from c×h×w to c×1×1, compress the number of channels to 1 / r times the original through the MLP module, expand to the original number of channels through the ReLU activation function, add it to the original feature map, get the channel attention output result through the sigmoid activation function, and multiply it with the original image to generate the intermediate feature map (c represents the number of feature map channels, r represents the parameter for adjusting the number of channels). The weights W0 and W1 of the MLP module are shared for the input. The channel attention mechanism is expressed as follows:

[0093] M c (F)=σ(MLP(AvgPool(Fm))+MLP(MaxPool(Fm)))

[0094] Where M c (F) represents the channel attention mechanism, σ represents the sigmoid activation function; W0 and W1 represent weight coefficients, W0∈R c / r×c, W1∈R c×c / r .

[0095] S4.2. Input the feature map in the spatial dimension, apply average pooling and maximum pooling operations to the feature map F output by the channel attention module, and the new feature map is F S avg and F S max , with a size of 1×h×w, and then the two operation results are concatenated. Finally, the convolution operation feature map is applied to generate a spatial attention feature map of size R×h×w.

[0096] The spatial attention mechanism is expressed as follows:

[0097] M s (F) = σ(f 7×7 ([AvgPool(Fm);MaxPool(Fm)]))

[0098] Where M s (F) represents the spatial attention mechanism; f 7×7 Represents the convolution process with a convolution kernel size of 7×7.

[0099] S5. The regression loss function CIoU is replaced by the WIoU loss function, and the fused features are passed into the detection head for classification prediction.

[0100] During the training process, it is inevitable to include low-quality samples. Geometric metrics such as distance and aspect ratio will intensify the penalty for low-quality instances, reducing the generalization performance of the model. When the overlap between the anchor box and the target box is high, the model should weaken the gradient penalty of the geometric metric and reduce the gradient penalty of low-quality samples, so that the target is focused on common samples to improve the generalization ability of the model.

[0101] In the regression loss calculation, the original network uses CIoU loss without considering geometric metrics such as distance and aspect ratio, which will aggravate the penalty for low-quality instances and reduce the generalization performance of the model. To address this problem, a new loss function WIoU is introduced to replace the original network loss function CIoU. By weighting the area between the predicted box and the true box, when the target box and the true box have a high overlap, the gradient penalty of the geometric metric is weakened, and the gradient penalty of low-quality samples is reduced. This makes the target more focused and the model has better generalization. Assume that (x, y) is at the position (x t ,y t ), then the WIOU formula is as follows.

[0102]

[0103] Where θ represents the Euclidean distance between the center point of the predicted box boundary and the center point of the real box boundary, and w c 、h c The width and height of the minimum bounding rectangle of the predicted box and the true box.

[0104] S6. Training the YOLOv7-ESC network model: The aluminum profile surface defect dataset is divided into training set, validation set and test set according to the ratio of 4:1:1 and input into the YOLOv7-ESC network model for pre-training. A total of 200 rounds of training are conducted, and the pre-training weight with the smallest training loss value is taken for transfer learning. The aluminum profile surface defect dataset and the pre-training weight are combined for training, and Mosaic is used for data enhancement to obtain the optimal weight.

[0105] The specific steps include:

[0106] S6.1. Randomly divide the aluminum profile surface defect dataset into training set, validation set and test set in a ratio of 4:1:1; randomly generate a reference point in the mosaic canvas with coordinates (x, y), where represents the horizontal coordinate of the reference point and represents the vertical coordinate of the reference point. The reference point is randomly generated. The coordinates of the vertex in the upper right corner of the mosaic canvas are (0, 0);

[0107] S6.2, as shown in Figure 6(a). According to the generated mosaic canvas, the original image is placed at the upper left, upper right, lower left, and lower right positions of the mosaic canvas reference point according to a fixed size; the size of the mosaic canvas is generally fixed, which is twice the size of the original image, and the canvas is the image after mosaic data enhancement;

[0108] S6.3, according to the size transformation method of each image, correspond the mapping relationship to the image label;

[0109] S6.4, as shown in Figure 6(b). The mosaic canvas is spliced ​​based on the randomly generated horizontal and vertical coordinates of the reference points, and four original images are randomly selected for edge processing in the mosaic canvas. For example, when the upper right corner of the original image exceeds the upper right corner edge of the canvas, it is necessary to process the part of the image that exceeds the canvas and retain the image area within the canvas. Mosaic data enhancement splices the four images, and each defect in the image has its corresponding coordinates. After splicing the four images, a new image is obtained, and the coordinates corresponding to the defects in this image are also obtained, and then the new image is passed into the neural network for learning;

[0110] S6.5. Input the sorted data set into the YOLOv7-ESC network model for pre-training, and take the pre-training weight with the smallest training loss value for transfer learning. Combine the aluminum profile surface defect data set and the pre-training weight for training to obtain the optimal weight.

[0111] S7. Use the trained optimal weights to conduct tests and evaluate the test results, ultimately achieving automated and intelligent identification of surface defect categories of aluminum profiles.

[0112] When evaluating the detection results, the evaluation indicators include mean Average Precision (mAP), recall (R), model volume, calculation amount, parameter amount and frame number (Frames Per Second, FPS).

[0113] The average precision is the mean of the average precision (AP) of all categories, indicating the AP of each category. iIt is the area enclosed by the curve of the precision (Precision, P) and the corresponding recall (Recall, R) of the i-th category in the range of 0-1.

[0114] The calculation formulas for mAP, recall rate, and frames per second (FPS) are as follows.

[0115]

[0116] Among them, AP i It is the area enclosed by the curve of the precision (Precision, P) and the corresponding recall (Recall, R) of the i-th category in the range of 0-1, N represents the total number of categories, TP is the number of samples correctly classified as positive, and FN is the number of samples incorrectly classified as negative; S represents the time used to detect an image, in seconds.

[0117] Table 1 Comparison of detection results of different models

[0118]

[0119] As can be seen from Table 1, the YOLOv7-ESC model proposed by the method of the present invention has greatly improved the performance of the YOLOv3 model, with mAP@0.5 increased by 8.7%, the model volume decreased by 49.5%, the number of parameters decreased by 48.2%, and the FPS greatly improved. Compared with the YOLOv5 model, mAP@0.5 increased by 4.4%, the model volume decreased by 32.8%, and the PFS greatly improved. Compared with the original YOLOv7 model, the YOLOv7-ESC model can also meet the real-time detection requirements in FPS, with an increase of 5.01 in FPS, but the mAP of this method increased by 6.8%, and the model volume, number of parameters and amount of calculation were reduced by 16.7%, 16.9% and 23.4% respectively, so the model proposed by this method can better meet the needs of industrial real-time detection.

[0120] The aluminum profile surface defect detection results of the YOLOv7-ESC model are as follows Figure 5 As shown in the figure, there are ten types of defects, including dents, non-conductivity, scratches, orange peel, leakage, bruises, pits, convex powder, coating cracks, and dirty spots. The YOLOv7-ESC network has a higher confidence in defect detection and a higher detection accuracy. It also has a higher detection accuracy in other defect detection. The comprehensive results show that the algorithm of the present invention can better detect the neglected small target defect areas and can achieve better detection effects in industrial production defect detection.

Claims

1. Aluminum profile defect detection and recognition method based on YOLOv7-ESC, characterized by The following steps are involved: Step 1: Obtain the aluminum profile surface defect dataset, then preprocess the aluminum profile surface image and label to obtain an aluminum profile surface defect map with prominent defects, and finally establish the YOLOv7-ESC network framework; Step 2: Improve the ELAN module and replace the Conv convolution module in the backbone network ELAN module with the PCnov convolution module to extract the surface defects of aluminum profiles; Step 3: Reconstruct the SPPCSPC module and construct the residual pyramid pooling structure SPPCSPC-F. The features extracted by the backbone network are input into the residual pyramid pooling structure SPPCSPC-F. Step 4: Add the CBAM attention mechanism module between the backbone network and the feature fusion network; Step 5: The regression loss function CIoU is replaced by the WIoU loss function, and the features fused by the feature fusion network are passed to the detection head for classification prediction; Step 6: Train the YOLOv7-ESC network model. First, divide the aluminum profile surface defect dataset into training set, validation set and test set. Use Mosaic to further enhance the data during the training process. Then input it into the YOLOv7-ESC network model for pre-training, and take the pre-training weight with the smallest training loss value for transfer learning. Finally, combine the aluminum profile surface defect dataset and the pre-training weight for training to obtain the optimal weight. Step 7: Use the trained optimal weights to conduct tests and evaluate the test results, ultimately achieving automated and intelligent identification of aluminum profile surface defects.

2. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: In the step 1, firstly, the images of the produced aluminum profiles are collected and sorted to obtain a data set of surface defects of the aluminum profiles, and the defects include 10 categories: dents, non-conductivity, scratches, orange peel, bottom leakage, bruises, pits, convex powder, coating cracks, and dirty spots.

3. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 2 is characterized in that: In step 1, the aluminum profile surface image is data enhanced and preprocessed, the defect area is annotated using Labelme software, and a json file is generated to save the annotation information; then the aluminum profile surface defect image is data preprocessed to achieve the effects of denoising and image enhancement, and an aluminum profile surface defect map with prominent defects is obtained.

4. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: In step 1, the YOLOv7-ESC network framework structure includes: a backbone network, a feature fusion network of the neck, and a detection head; the backbone network extracts the surface defect features of the aluminum profile in the input image; first, the input feature image is 640×640×3, and the features are refined and extracted through four layers of CBL convolution blocks; then, the ELAN-P module reduces the number of parameters and downsamples to reduce the number of feature channels; then, the feature map of size 20×20×1024 is input to the neck feature fusion network, and the neck feature fusion network uses top-down and bottom-up connections to perform feature fusion processing on multi-scale feature layers; finally, it is sent to the detection head to output feature maps of different scales for loss calculation, defect location, and defect prediction.

5. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: In step 2, the 3*3Conv convolution module in the ELAN module is replaced by a PConv convolution module, which can reduce the number of module parameters; the ELAN-P module consists of a short path and a long path, the short path is a 1*1CBS convolution, and the long path consists of a 1*1CBS convolution and 4 consecutive PConv convolutions.

6. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 5 is characterized in that: Assume that the size of the input feature map X is c×h×w, where h, w, and c represent the height, width, and number of channels of the input feature map X, respectively; enter Characteristic Features I∈R c×h×w , the output is O∈R c×h×w , the standard convolution kernel size is w∈R c×k×k , k represents the convolution kernel size; since the PConv convolution module only needs to feature extract part of the channel cp, the convolution kernel size is reduced to w∈R cp×k×k ; The formula for the ratio of PConv convolution and Conv convolution calculation is as follows: In the above formula, FLOPS PConv Indicates PConv calculation amount; FLOPS Conv represents the amount of PConv calculation; h represents the height of the input feature map; w represents the width of the input feature map; c p represents the number of channels extracted by PConv; c represents the number of channels of the input feature map.

7. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: In the step 3, the residual pyramid pooling structure SPPCSPC-F includes an upper multi-scale feature extraction channel and a lower small target feature extraction channel; the upper multi-scale feature extraction channel captures feature information of different scales through different pooling kernels to obtain richer feature target information, and a 1×1 convolution adjustment channel is formed before and after the pooling layer; the lower small target feature extraction channel introduces three layers of residual convolution blocks, and three consecutive linear residual convolution blocks CF modules refine the extracted features to extract more small target features.

8. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: The step 4 comprises the following steps: S4.

1. Input the feature map in the channel dimension, pass through the parallel MaxPool layer and AvgPool layer, get two 1×1 channel weight matrices, change the feature map dimension from c×h×w to c×1×1, compress the number of channels to 1 / r times of the original through the MLP module, expand to the original number of channels through the ReLU activation function, add it to the original feature map, get the channel attention output result through the sigmoid activation function, and multiply it with the original image to generate the intermediate feature map; the weights W0 and W1 of the MLP module are shared for the input; the channel attention mechanism is expressed as follows: M c (F)=σ(MLP(AvgPool(Fm))+MLP(MaxPool(Fm)) Where M c (F) represents the channel attention mechanism; σ represents the sigmoid activation function; W0 and W1 represent weight coefficients, W0∈R c / r×c , W1∈R c×c / r ; AvgPool(Fm) represents the feature map processed by the average pooling layer; MaxPool(Fm) represents the feature map processed by the maximum pooling layer; MLP(AvgPool(Fm)) represents the feature map processed by the average pooling layer and the MLP module; MLP(MaxPool(Fm)) represents the feature map processed by the maximum pooling layer and the MLP module; S4.

2. Input the feature map in the spatial dimension, apply average pooling and maximum pooling operations to the feature map F output by the channel attention module, and the new feature map is denoted as F S avg and F S max , F S avg represents the feature map of channel dimension average pooling, F S max Represents the feature map of the maximum pooling of the channel dimension, with a size of 1×h×w; then the new feature maps obtained by the two pooling operations are connected; finally, the convolution operation feature map is applied to generate a spatial attention feature map of size R×h×w; The spatial attention mechanism is expressed as follows: M s (F)=σ(f 7×7 ([AvgPool(Fm);MaxPool(Fm)])) Where M s (F) represents the spatial attention mechanism, f 7×7 Represents the convolution process with a convolution kernel size of 7×7.

9. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1 is characterized in that: In step 5, the regression loss function CIoU is replaced by the WIoU loss function, and the features fused by the feature fusion network are passed to the detection head for classification prediction; The WIoU loss function formula is as follows: Where α is a hyperparameter; β is the outlier degree; θ represents the Euclidean distance between the center point of the predicted box boundary and the center point of the true box boundary; r represents the gradient gain; w c 、h c L are the width and height of the minimum bounding rectangle of the predicted box and the real box respectively; WIOU represents the WIoU loss, L IOU represents the intersection-over-union loss; x represents the horizontal coordinate of the center point of the prediction box, y represents the vertical coordinate of the center point of the prediction box, and x t Indicates the horizontal coordinate of the center point of the target frame, y t Indicates the vertical coordinate of the center point of the target box.

10. The aluminum profile defect detection and identification method based on YOLOv7-ESC according to claim 1, characterized in that: The step 6 comprises the following steps: S6.

1. The aluminum profile surface defect dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 4:1:

1. A reference point is randomly generated in the mosaic canvas with the coordinates (x c ,y c ), represents the horizontal coordinate of the reference point, represents the vertical coordinate of the reference point; the reference point is randomly generated; the coordinates of the upper right corner of the mosaic canvas are (0, 0); S6.

2. Based on the generated mosaic canvas, the original image is placed at the upper left, upper right, lower left, and lower right positions of the mosaic canvas reference point according to a fixed size. The size of the mosaic canvas is generally fixed, which is twice the size of the original image. The canvas is the image after mosaic data enhancement. S6.3, according to the size transformation method of each image, correspond the mapping relationship to the image label; S6.4, stitching the mosaic canvas according to the randomly generated horizontal and vertical coordinates of the reference points, and randomly selecting four original images to perform edge processing in the mosaic canvas; for example, when the upper right corner of the original image exceeds the upper right corner edge of the canvas, it is necessary to process the part of the image that exceeds the canvas and retain the image area within the canvas; S6.

5. The sorted data set is input into the YOLOv7-ESC network model for pre-training, and the pre-training weight with the smallest training loss value is taken for transfer learning; the aluminum profile surface defect data set and the pre-training weight are combined for training to obtain the optimal weight.

Images