Aluminum strip surface defect detection method, system and device based on lightweight model

Abstract

The application discloses an aluminum strip surface defect detection method, system and equipment based on a lightweight model, and relates to the field of surface defect detection and machine vision. The surface defect segmentation model constructed by the method is a lightweight model composed of a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network and a prediction network. While maintaining high segmentation accuracy, the model parameter quantity and complexity are greatly reduced, and the aluminum strip surface defect detection efficiency, accuracy and reliability are improved. The application lays a theoretical foundation for the deployment of the aluminum strip surface defect detection algorithm on mobile terminals and embedded devices, and has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of surface defect detection and machine vision in industrial scenarios, and particularly relates to an aluminum strip surface defect detection method, system and equipment based on a lightweight model. BACKGROUND

[0002] Aluminum strip is widely used in electronic products, packaging containers, new energy vehicles and other fields due to its good mechanical properties and low cost. However, due to factors such as aging of rolling equipment, dust, and unreasonable process parameters, the aluminum strip surface produces various defect types such as holes, black spots, and scratches during the production process, which directly affects the appearance of subsequent finished and semi-finished products, and even affects the safety and performance of the products. Therefore, the detection of aluminum strip surface defects has become an indispensable key link in actual production. At present, the detection of aluminum strip surface defects on the production line is mainly based on manual visual observation. This method not only has low efficiency and high labor cost, but also is seriously affected by the subjective factors of workers in defect judgment, and the detection accuracy cannot be guaranteed.

[0003] In addition, the traditional defect detection method based on machine learning selects and extracts defect features through artificial image processing, which is difficult to extract all features of complex surfaces and has poor generalization ability. In recent years, the convolutional neural network (CNN) based on deep learning has solved the above problems. The network gradually extracts complex high-level feature information from the image through multiple convolutional layers, has strong representation learning ability, and provides a new solution for aluminum strip surface defect detection. The target detection network YOLO (You Look Only Once) series and R-CNN series based on CNN have been widely used in the recognition and positioning of complex surface defects of industrial products. However, there are still some problems in the detection of surface defects in the production process of aluminum strip that need to be solved. On the one hand, some defects on the surface of aluminum strip, such as scratches and color differences, appear in the form of strips with changing angles. In this case, the anchor box (Bounding box) in the YOLO series and the R-CNN series has limited effect, as it contains a large area of normal regions without defects. On the other hand, real-time performance is a necessary requirement for surface defect segmentation in the production process of aluminum strip. Under the premise of not affecting the production speed, the visual detection system arranged on the production line should complete the real-time segmentation of defects, which requires the network to have high inference efficiency and small model size to achieve the optimal balance of segmentation speed, accuracy and reliability. On the other hand, due to the complex production environment of aluminum strip, the surface defects have significant size difference, inter-class similarity and intra-class difference, which brings great challenges to the network for accurate defect boundary segmentation and recognition. This requires the network to extract more rich feature information, learn more comprehensive shallow detail expression and deep semantic expression, and thus improve the accuracy of defect segmentation. SUMMARY

[0004] In order to solve the problems in the background art, the present application provides an aluminum strip surface defect detection method, system and equipment based on a lightweight model, which greatly reduces the calculation amount and parameter amount of the model, and improves the efficiency, accuracy and reliability of aluminum strip surface defect detection.

[0005] In order to achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] On the one hand, the present application provides an aluminum strip surface defect detection method based on a lightweight model, comprising:

[0007] Real-time acquisition of original image data of the surface of the aluminum strip by an industrial line scan camera;

[0008] The original image data is evenly divided into a plurality of square subgraphs, and the subgraphs containing defects are screened out;

[0009] Pixel-level labeling of the screened defect subgraphs, marking the defect contours and categories, to obtain a surface defect dataset; the surface defect dataset includes a training set, a validation set and a test set;

[0010] Data augmentation of the surface defect dataset to obtain a data-augmented surface defect dataset;

[0011] Construction of a surface defect segmentation model, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network and a prediction network;

[0012] Training, validation and testing of the surface defect segmentation model using the data-augmented surface defect dataset to obtain a trained surface defect segmentation model;

[0013] Real-time detection of surface defects in the production process of the aluminum strip using the trained surface defect segmentation model.

[0014] Optionally, the pixel-level labeling of the screened defect subgraphs, marking the defect contours and categories, to obtain a surface defect dataset, specifically comprises:

[0015] Using LabelMe to manually classify and pixel-level segmentation label the defects in the screened defect subgraphs, marking the defects of different categories with different colors to form a JSON format label dataset;

[0016] Converting the JSON format label dataset into a PNG format label dataset;

[0017] 20% of each type of defect subgraph and its corresponding PNG format image are randomly divided into a test set, 90% of the remaining images are divided into a training set, and the last 10% is a validation set;

[0018] The test set, the training set and the validation set composed of different categories of defect subgraphs and their corresponding PNG format images are integrated respectively, and each data set after integration is made into the same format as the VOC2007 data set, so as to obtain the surface defect data set.

[0019] Optionally, the surface defect data set is subjected to data enhancement to obtain a data-enhanced surface defect data set, specifically including:

[0020] The surface defect data set is subjected to data enhancement by using one or more data enhancement methods of random cropping, random horizontal flipping, random vertical flipping, scale jittering, color jittering or Mosaic, to obtain a data-enhanced surface defect data set.

[0021] Optionally, the surface defect segmentation model is constructed, specifically including:

[0022] A joint loss function of the surface defect segmentation model is constructed;

[0023] A backbone feature extraction network is built based on a MobileViTv2 network and a CBAM attention module;

[0024] A multi-scale feature fusion network is built based on an SPPF module and an HRFPN network;

[0025] A deep supervision network is established based on a convolutional layer and the joint loss function;

[0026] A prediction network is constructed based on a convolutional layer;

[0027] The backbone feature extraction network, the multi-scale feature fusion network, the deep supervision network and the prediction network are connected to form the surface defect segmentation model.

[0028] Optionally, the surface defect segmentation model is trained, validated and tested using the data-enhanced surface defect data set to obtain a trained surface defect segmentation model, specifically including:

[0029] The surface defect segmentation model is trained using the data-enhanced training set, and the corresponding weight file in the training process is saved;

[0030] Each weight file is tested on the data-enhanced validation set, and the hyperparameters of the surface defect segmentation model are adjusted, and the weight file with the highest segmentation accuracy is selected as the model parameter of the trained surface defect segmentation model;

[0031] The data-enhanced test set is input into the trained surface defect segmentation model to view the approximate generalization performance of the model.

[0032] In another aspect, the present application provides a lightweight model-based aluminum strip surface defect detection system, comprising:

[0033] An original image acquisition module is configured to acquire original image data of the surface of the aluminum strip in real time using an industrial line scan camera.

[0034] An image cropping and screening module is configured to divide the original image data into a plurality of square sub-images and screen out sub-images containing defects.

[0035] An image labeling module is configured to perform pixel-level labeling on the screened defect sub-images, mark the defect contours and categories, and obtain a surface defect dataset; the surface defect dataset includes a training set, a validation set, and a test set.

[0036] A data enhancement module is configured to perform data enhancement on the surface defect dataset to obtain a data-enhanced surface defect dataset.

[0037] A model construction module is configured to construct a surface defect segmentation model, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network, and a prediction network.

[0038] A model training module is configured to train, validate, and test the surface defect segmentation model using the data-enhanced surface defect dataset to obtain a trained surface defect segmentation model.

[0039] A surface defect detection module is configured to use the trained surface defect segmentation model to detect surface defects in real time during the production of the aluminum strip.

[0040] In another aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the lightweight model-based aluminum strip surface defect detection method.

[0041] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed to implement the lightweight model-based aluminum strip surface defect detection method.

[0042] According to the embodiments of the present application, the following technical effects are achieved:

[0043] The aluminum strip surface defect detection method, system and equipment based on a lightweight model provided by the application utilize an industrial line scan camera to collect original image data of the surface of an aluminum strip in real time; the original image data is evenly divided into a plurality of square subgraphs, and subgraphs containing defects are screened out; the screened defect subgraphs are pixel-level labeled to mark defect contours and categories, and a surface defect dataset is obtained; the surface defect dataset includes a training set, a verification set and a test set; the surface defect dataset is subjected to data enhancement to obtain a data-enhanced surface defect dataset; a surface defect segmentation model is constructed, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network and a prediction network; the data-enhanced surface defect dataset is used to train, verify and test the surface defect segmentation model, and a trained surface defect segmentation model is obtained; the trained surface defect segmentation model is used to detect surface defects in the production process of the aluminum strip in real time. The method can greatly reduce the calculation amount and parameter amount of the model, and improve the efficiency, accuracy and reliability of aluminum strip surface defect detection. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A flowchart of an aluminum strip surface defect detection method based on a lightweight model provided by the present application;

[0046] Figure 2 An image acquisition, cropping, screening and labeling process diagram in the method of the present application;

[0047] Figure 3 A structure diagram of a surface defect segmentation model proposed by the present application;

[0048] Figure 4 A structure diagram of a lightweight feature aggregation node proposed by the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] The application aims to provide an aluminum strip surface defect detection method based on a lightweight model, which greatly reduces the calculation amount and parameter amount of the model and improves the aluminum strip surface defect detection efficiency, accuracy and reliability.

[0051] In order to make the above-mentioned purposes, characteristics and advantages of the application more obvious and easy to understand, the application will be further described in detail below in combination with the drawings and specific embodiments.

[0052] Figure 1 A flowchart of an aluminum strip surface defect detection method based on a lightweight model provided by the application is shown in Figure 1 An aluminum strip surface defect detection method based on a lightweight model comprises the following steps.

[0053] Step 1: Real-time acquisition of original image data of the aluminum strip surface by using an industrial line scan camera.

[0054] The aluminum strip surface image acquisition in the aluminum strip production process is to use an industrial line scan camera and an LED light source to real-time acquire original image data of the fast-moving aluminum strip surface, as shown in Figure 2 The acquired original image pixels are 4096x1024, and the length (4096 pixels) corresponds to the actual width of the aluminum strip.

[0055] Specifically, an industrial line scan camera and a white LED parallel light source are equipped on the production line. Then, the encoder is adjusted to trigger the camera to shoot the aluminum strip at a frequency corresponding to the production line speed, and the original pictures are stored in an SD card.

[0056] Step 2: Uniformly dividing the original image data into multiple square subgraphs and screening out subgraphs containing defects.

[0057] Step 2 performs image cropping and screening of the original image, as shown in Figure 2 The original image shot in step 1 is uniformly divided into several low-resolution square subgraphs, and the subgraphs without defects are removed to reduce the model calculation amount, and then the subgraphs containing defects are screened out as defect subgraphs in subsequent step 3.

[0058] Step 2 screens not less than 600xn (n is the number of defect categories) subgraphs containing defects to establish a data set, and the number of images of each category of defects is not less than 600. The 5 common defects include color difference, edge crack, pit, black spot and scratch. During the screening process, sample balancing is performed on each category of defect pictures to make the number of pictures of each category of defects approximately the same.

[0059] Step 3: Pixel-level labeling of the screened defect subgraphs, marking the defect contour and category, and obtaining a surface defect data set.

[0060] Step 3: Pixel-level labeling is performed on the defect subgraphs screened in step 2 to mark the defect contours and categories, and a surface defect dataset is obtained, specifically including:

[0061] Step 3.1: Artificial classification and pixel-level segmentation labeling are performed on the defects in the screened defect subgraphs using LabelMe, and different categories of defects are marked with different colors to form a JSON format defect picture label dataset;

[0062] Step 3.2: The JSON format label dataset is converted into a PNG format label dataset, and the PNG file naming is one-to-one corresponding to the original picture name;

[0063] Step 3.3: 20% of each category of defect subgraph and its corresponding PNG format image are randomly divided into a test set, and 90% of the remaining images are divided into a training set, and the last 10% is a validation set;

[0064] Step 3.4: The test set, training set and validation set composed of different category defect subgraphs and their corresponding PNG format images are integrated respectively, and the integrated datasets are made into the same format as the VOC2007 dataset, so as to obtain the surface defect dataset.

[0065] Therefore, the surface defect dataset includes a training set, a validation set and a test set in the same format as the VOC2007 dataset. The training set is used to train the model and determine the parameters, and has the largest number. The validation set is used to detect the performance of the model, and the model parameters can be adjusted according to the test results, mainly the hyperparameters. The test set is used to detect the generalization performance of the final trained model, and the model parameters will not be changed after the test set.

[0066] If there are only training set and test set, the model cannot be evaluated before the test set. In fact, it is usually more desirable to evaluate the model before determining the final model, then adjust the model parameters (especially the hyperparameters) according to the evaluation results, and retrain until the model is basically trained, and finally test with the test set. The test is to check the approximate generalization performance of the model, and the model parameters will not be updated. It is for this purpose that the present application divides a validation set. The validation set is still a training set to some extent, because the model parameters will be adjusted according to the results of the validation set.

[0067] Step 4: Data augmentation is performed on the surface defect dataset to obtain a data-augmented surface defect dataset.

[0068] The surface defect data set in step 3 is subjected to data enhancement, and the data enhancement manner includes one or a combination of the following data enhancement manners, that is, random cropping, random horizontal flipping, random vertical flipping, scale jittering, color jittering or Mosaic, so as to obtain a data-enhanced surface defect data set. When data enhancement is performed, it is necessary to ensure that the defects in the image still match the annotation boundary after image transformation.

[0069] Step 5: constructing a surface defect segmentation model, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network and a prediction network.

[0070] The surface defect segmentation model construction process specifically includes:

[0071] Step 5.1: constructing a joint loss function of the surface defect segmentation model.

[0072] The present application adopts a Dice Loss function and a binary cross-entropy loss function to construct a joint loss function L Union , which is expressed as follows: Wherein and y respectively represent the segmentation result predicted by the model and the artificial annotation; ε is a small value, which is used to avoid sharp fluctuations in the value; N is the number of batches of images input into the model (batch size), and y i respectively represent the model-predicted segmentation result and the artificial annotation of the i-th image in each batch of images; and β are parameters for balancing the proportion of the two loss functions.

[0073] Step 5.2: constructing a backbone feature extraction network based on a MobileViTv2 network and a CBAM attention module.

[0074] The MobileViTv2 network adopted is a lightweight feature extraction network, and the improvement of the present application is that the last average pooling layer and the fully connected layer of the network are removed, and ten layers are composed of a 3x3 ordinary convolution layer, six MobileNetv2 block layers and three MobileViTv2 block layers; the ten layers of the MobileViTv2 network are divided into the following five basic blocks (block), and the first layer in each basic block is subjected to a down-sampling operation. Among them,

[0075] Basic block one: 3x3Conv(stride=2)+MobileNetv2 block(stride=1);

[0076] Basic block two: MobileNetv2 block (stride=2) + MobileNetv2 block (stride=1);

[0077] Basic block three: MobileNetv2 block (stride=2) + MobileViTv2 block;

[0078] Basic block four: MobileNetv2 block (stride=2) + MobileViTv2 block;

[0079] Basic block five: MobileNetv2 block (stride=2) + MobileViTv2 block.

[0080] The above five basic blocks are directly connected, that is, the output features of a basic block are taken as the input of the next basic block. Another improvement of the present application is that a CBAM attention module is embedded after each basic block, and the output features of each basic block are input into the next basic block after being processed by the CBAM attention module. At the same time, the five different scale features output after CBAM processing are directly input into the multi-scale feature fusion network, as shown in Figure 3 The five features are first input into five different SPPF modules in the multi-scale feature fusion network.

[0081] The present application embeds a CBAM attention module after each basic block of the MobileViTv2 network, and the whole network is named CBAM-MobileViTv2 as the backbone feature extraction network of the present application. Specifically, the CBAM attention module is composed of a channel attention module and a spatial attention module, wherein the channel attention module is composed of a spatial dimension average pooling layer and a maximum pooling layer, a multi-layer perceptron, and a sigmoid activation function, and the spatial attention module is composed of a channel dimension average pooling layer and a maximum pooling layer, and a sigmoid activation function.

[0082] The improved CBAM-MobileViTv2 network of the present application outputs five scale feature maps after five times of down-sampling, which are P1, P2, P3, P4, and P5, as shown in Figure 3 The scales are 256x256x48, 128x128x96, 64x64x192, 32x32x288, and 16x16x384, respectively.

[0083] Step 5.3: Building a multi-scale feature fusion network based on SPPF module and HRFPN network.

[0084] The multi-scale feature fusion network of the present application is composed of an SPPF module and an HRFPN network. The SPPF module is composed of a convolution layer with a step of 1 and a convolution kernel of 1x1 and three 5x5 maximum pooling layers connected in series. The output features of each layer are fused by a concatenation operation and the channel number is adjusted by another 1x1 convolution layer. A batch normalization layer and a SiLU layer are added after each 1x1 convolution layer.

[0085] The HRFPN network is composed of four gradually shallower top-down branches. The number of paths gradually increases in width, and the number of nodes in the depth direction of each path gradually decreases as the number of paths increases.

[0086] The HRFPN network uses lightweight feature aggregation nodes, such as Figure 4 As shown, each node has two or three edges to input feature maps with different resolutions (including low-level feature maps, current feature maps, and high-level feature maps). They are adjusted to the same resolution after down-sampling or up-sampling operations. Then, these feature maps are integrated by element-wise summation and feature fusion is performed by a MobileNetv2 block with a step of 1 (stride=1). Figure 4 where W and H represent the width and height of the feature map respectively, different subscripts correspond to different feature maps, and subscripts l, c, and h correspond to low-level feature maps, current feature maps, and high-level feature maps, respectively. For example, W c and H c represent the width and height of the current feature map; C represents the number of channels of the feature map, and the number of channels of the feature map in each node is the same.

[0087] The HRFPN network uses fast normalization fusion weighted features, which can assign different weights to input features of different scales. This method is represented as Different resolutions of input features contribute unequally to the output features, so it is necessary to apply additional weights to each input feature to represent its importance. In the formula, w i is a learnable multi-dimensional tensor weight, ∈=0.0001 is used to avoid the instability of numerical fluctuations, I i represents the input feature map, the value of the normalized weight is between 0 and 1, and O represents the feature map after using the normalized weight.

[0088] The feature maps P1, P2, P3, P4 and P5 output by the backbone feature extraction network are sent into the HRFPN network after being processed by the SPPF module, and four feature maps Q1, Q2, Q3 and Q4 of the same resolution are output.

[0089] Step 5.4: Establishing a deep supervision network based on a convolutional layer and the joint loss function.

[0090] Referring to Figure 3 The feature maps Q1, Q2, Q3 and Q4 output by the multi-scale feature fusion network are subjected to two times of upsampling to make their resolutions consistent with that of the input image, and the number of channels of the feature maps is adjusted to n+1 (background) through a convolutional layer (Conv) with a step of 1 and a convolution kernel of 1x1. The joint loss function L Union The four feature maps output by the deep supervision network are respectively subjected to upsampling to make their resolutions consistent with that of the input image, and the number of channels of the feature maps is adjusted to n+1 (background) through a convolutional layer (Conv) with a step of 1 and a convolution kernel of 1x1. Loss calculation is performed with the artificial annotation y, and four joint loss function calculation results Loss1-4 are obtained. Loss calculation is performed on feature maps of different resolutions, which can make full use of the detailed information of high resolution and the semantic abstract information of low resolution, promote the flow of information within the model, and accelerate the network optimization process.

[0091] Step 5.5: Establishing a prediction network based on a convolutional layer.

[0092] As shown in Figure 3 The four feature maps output by the deep supervision network are subjected to upsampling to make their resolutions consistent with that of the input image, and the number of channels of the feature maps is adjusted to n+1 (background) through a convolutional layer (Conv) with a step of 1 and a convolution kernel of 1x1. The feature map is not only used as the output of the final prediction network

[0093] Step 5.6: Connecting the backbone feature extraction network, the multi-scale feature fusion network, the deep supervision network and the prediction network to constitute the surface defect segmentation model.

[0094] Therefore, the surface defect segmentation model constructed by the present application adopts CBAM-MobileViTv2 to extract defect features, adopts SPPF modules and HRFPN networks for multi-scale feature fusion, adopts a deep supervision network to accelerate the convergence process of the network, and adopts a prediction network to output the final defect prediction result, thereby realizing a lightweight model capable of segmenting n different types of defects, having high segmentation accuracy, strong generalization ability, small network model, and fast segmentation speed, and capable of being quickly deployed on embedded devices and meeting the requirements of real-time defect segmentation on an aluminum strip production line.

[0095] Step 6: training, verifying and testing the surface defect segmentation model by using the data-enhanced surface defect dataset to obtain a trained surface defect segmentation model.

[0096] The training, verifying and testing process of the surface defect segmentation model specifically includes:

[0097] Step 6.1: training the surface defect segmentation model by using the data-enhanced training set and saving the corresponding weight file in the training process.

[0098] The corresponding weight file in the training process refers to the weight file of the model parameters generated by the model through continuous iterative calculation in the training process in order to minimize the loss function between the real label and the prediction result. Therefore, the weight file is the parameter of the segmentation model obtained by training, and many weight files will be generated with the iterative calculation in the training process, and the loss function will be continuously reduced, and the prediction of the weight file for defects will also be more and more accurate.

[0099] Step 6.2: loading each weight file into the surface defect segmentation model, testing on the data-enhanced validation set, adjusting the hyperparameters of the surface defect segmentation model, and selecting the weight file with the highest segmentation accuracy as the model parameter of the trained surface defect segmentation model.

[0100] The segmentation accuracy is generally represented by the average intersection over union (mIoU, mean intersection over union), that is, the average value of the ratio of the intersection and union of all categories, as follows:

[0101]

[0102] where p ij represents the number of pixels of the i-th category predicted as the j-th category, p ji represents the number of pixels of the j-th category predicted as the i-th category, p ii represents the number of pixels of the i-th category correctly predicted as the i-th category, and k represents the number of defect categories.

[0103] Step 6.3: input the data enhanced test set into the trained surface defect segmentation model to view the approximate generalization performance of the model.

[0104] Step 7: use the trained surface defect segmentation model to detect surface defects in the aluminum strip production process in real time.

[0105] When performing surface defect segmentation prediction, only the picture to be tested needs to be input into the trained surface defect segmentation model (i.e. the model after loading the weight file with the highest segmentation accuracy), and the defect category, defect position and boundary condition (defect contour) can be output, realizing real-time detection of surface defects in the aluminum strip production process.

[0106] Compared with the detection method in the prior art, the present application has the following advantages: a lightweight surface defect segmentation model is constructed, an improved MobileViTv2 is used as a feature extraction network, a CBAM attention mechanism is embedded after each basic block of the MobileViTv2 to enrich the feature information extracted by the network, an SPPF module is used in the multi-scale feature fusion network to strengthen the connection of local and global information in different scale feature maps, an HRFPN network is used to fully fuse shallow detail information on the basis of preserving high-level semantic information, and the network can accurately segment the boundaries of defects with large differences in morphology, size and complex types; in the deep supervision network, the output feature maps of the five branches are calculated by the loss function, so that the network is more fully trained and the convergence process of the network is accelerated; the segmentation result outputs not only the defect category information and position information, but also the accurate contour and boundary of the defect, and then calculates the area and density of the defect, which helps enterprises to obtain more detailed defect information and make more accurate judgments on the quality of aluminum strip, and the application prospect is very broad.

[0107] A specific embodiment of the aluminum strip surface defect detection method based on the lightweight model of the present application is provided below, including the following steps:

[0108] S1: aluminum strip surface image acquisition, using an industrial line scan camera and an LED light source, real-time acquisition of original image data of fast moving aluminum strip surface, the original image pixel is 4096x1024;

[0109] When collecting images, an industrial line scan camera and a white LED parallel light source are provided on the production line, the speed of the aluminum strip moving on the production line is 50m / min, the encoder is adjusted to trigger the camera to shoot the aluminum strip at a frequency corresponding to the speed of the production line, and the original photo is stored in an SD card.

[0110] S2: Image cropping and screening, the original image taken in S1 is uniformly divided into 16 low-resolution square sub-images, each sub-image has a resolution of 512x512, and the sub-images containing defects are screened out; a total of 3852 sub-images including defects are screened out to establish a data set, including 5 common defects: color difference, edge cracking, pit, black spot and scratch. During the selection process, sample balancing is performed on each type of defect image to make the number of each type of defect image approximately the same.

[0111] S3: Image annotation, pixel-level annotation is performed on the defect sub-images screened out in S2, the defect contour and category are marked, and a surface defect data set is obtained;

[0112] LabelMe is used to manually classify and pixel-level contour label the defects in the image, different categories of defects are marked with different colors, and a JSON format defect image label data set is formed. The JSON format label data set is converted into a PNG format label data set, and the PNG file naming is one-to-one corresponding to the original image name; 20% of the PNG images corresponding to each category of defect image are randomly divided into a test set, and the remaining 90% of the images are divided into a training set, and the last 10% is a validation set. The data of different categories of defects are integrated to form the final test set, training set and validation set, the specific number is 2768 for the training set, 312 for the validation set, and 772 for the test set. The data set is made into the same format as the VOC2007 data set to obtain a surface defect data set.

[0113] S4: Data augmentation, the surface defect data set is the basis for defect detection. Considering that there is less defect data in the industry, the surface defect data set described in S3 is subjected to data augmentation. Data augmentation techniques mainly include random cropping, random horizontal flipping, random vertical flipping, scale jittering, color jittering or Mosaic, etc. At the same time of data augmentation, it is ensured that the defects in the image still match the annotation boundary after image transformation.

[0114] S5: Surface defect segmentation model construction, the model uses CBAM-MobileViTv2 to extract defect features, SPPF module and HRFPN network for multi-scale feature fusion, deep supervision network to accelerate the convergence process of the network, and prediction network to output the final defect prediction result; wherein:

[0115] S5.1: Constructing the loss function of the segmentation network, using Dice Loss function and binary cross-entropy loss function Construct a joint loss function L Union , as follows: wherein and y respectively represent the segmentation results predicted by the model and the manual annotation; ε is a small value to avoid dramatic fluctuations in values; N is the number of images in each batch input to the model (batchsize), and y i respectively represent the segmentation results predicted by the model and the manual annotation for the i-th image in each batch; and β are parameters for balancing the proportion of the two loss functions, both of which are set to 0.5.

[0116] S5.2: Build a backbone feature extraction network based on the MobileViTv2 network and the CBAM attention module;

[0117] The MobileViTv2 network is a lightweight feature extraction network, and the MobileViTv2 network of the application is composed of one 3x3 ordinary convolution layer, six MobileNetv2 block layers and three MobileViTv2 block layers, a total of ten layers, and the last average pooling layer and the full connection layer of the network are removed; the ten layers of the MobileViTv2 network are divided into the following five basic blocks, and the first layer in each basic block is subjected to a down-sampling operation; wherein,

[0118] Basic block one: 3x3Conv(stride=2)+MobileNetv2 block(stride=1);

[0119] Basic block two: MobileNetv2 block(stride=2)+MobileNetv2 block(stride=1);

[0120] Basic block three: MobileNetv2 block(stride=2)+MobileViTv2 block;

[0121] Basic block four: MobileNetv2 block(stride=2)+MobileViTv2 block;

[0122] Basic block five: MobileNetv2 block(stride=2)+MobileViTv2 block;

[0123] Each basic block of the MobileViTv2 network is embedded with a CBAM attention module (Convolutional block attention module), and the entire network is named CBAM-MobileViTv2. The CBAM attention mechanism is composed of a channel attention module and a spatial attention module, wherein the channel attention module is composed of a spatial dimension average pooling layer and a maximum pooling layer, a multi-layer perceptron (MLP), and a sigmoid activation function, and the spatial attention module is composed of a channel dimension average pooling layer and a maximum pooling layer, and a sigmoid activation function. The improved CBAM-MobileViTv2 network is subjected to five times of down-sampling, and five scale feature maps P1, P2, P3, P4 and P5 are correspondingly output.

[0124] S5.3: Building a multi-scale feature fusion network based on the SPPF module and the HRFPN network;

[0125] The multi-scale feature fusion network is composed of an SPPF module (Spatial Pyramid Pooling Fast) and an HRFPN network (High Resolution Feature Pyramid Network).

[0126] The SPPF module is composed of a convolution layer with a step of 1 and a convolution kernel of 1x1 and three 5x5 maximum pooling layers connected in series. The output features of each layer are fused by a concatenation operation and are subjected to channel number adjustment by another 1x1 convolution layer. A batch normalization (Batchnormalization) layer and a SiLU layer are added after each 1x1 convolution layer.

[0127] The HRFPN network is composed of four gradually shallower top-down branches. The network gradually increases the number of paths in the width direction, and the number of nodes in the depth direction of each path gradually decreases as the number of paths increases.

[0128] The HRFPN network uses a lightweight feature aggregation node as shown in Figure 3 Each node has two or three edges to input feature maps with different resolutions. Low-level feature maps are down-sampled by a Depthwise convolution with a step of 2, and high-level feature maps are up-sampled by a nearest neighbor interpolation method. Different feature maps are adjusted to the same resolution. Then, these feature maps are integrated by element-wise summation and subjected to feature fusion by a MobileNetv2 block with a step of 1.

[0129] The HRFPN network uses fast normalization fusion weighting features, which can assign different weights to input features of different scales. The method is represented as where w i is a learnable multi-dimensional tensor weight, ∈ = 0.0001 is used to avoid the instability of numerical fluctuations, I i represents the input feature map, the value of the normalized weight is between 0 and 1, and O represents the feature map after using the normalized weight.

[0130] The feature maps P1, P2, P3, P4, and P5 output by the backbone feature extraction network are sent to the HRFPN network after being processed by the corresponding SPPF modules, and four feature maps Q1, Q2, Q3, and Q4 of the same resolution are output.

[0131] S5.4: Build a deep supervision network;

[0132] The feature maps Q1, Q2, Q3, and Q4 are twice up-sampled to have the same resolution as the input image, and the number of channels of the feature maps is adjusted to 6 (5 defects + background) through a convolution layer with a step of 1 and a convolution kernel of 1x1. The joint loss function L Union The four output feature maps are respectively loss calculated with artificial labeling.

[0133] S5.5: Build a prediction network;

[0134] The four feature maps output by the deep supervision network are spliced, and the number of channels of the obtained feature map is adjusted to 6 (5 defects + background) through a convolution layer with a step of 1 and a convolution kernel of 1x1. This feature map is not only used as the output of the final prediction network, but also is also loss calculated.

[0135] S6: Surface defect segmentation model training, save the corresponding weight file in the training process, test each weight file on the validation set to adjust the hyperparameters of the model, select the weight with the highest segmentation accuracy as the final trained model parameter, and input the test set picture into the final model to view the approximate generalization performance of the model;

[0136] In S6, network training combines transfer learning and network fine-tuning. Pre-trained weights are loaded before training begins. The entire training process is divided into two phases totaling 200 epochs. The first phase, consisting of the first 50 epochs, freezes the backbone feature extraction network, with a batch size of 8 images. The second phase, comprising the remaining 150 epochs, unfreezes the network, and sets the batch size to 4. A warmup strategy is used for the first 5 training epochs, followed by a cosine learning plan. The Adam optimizer is used, with an initial learning rate set to 5e. -4 The exponential decay factor β1 was set to 0.9 and β2 was set to 0.999.

[0137] S7: Surface defect segmentation prediction. Input the image to be tested into the model and load the weights trained in S6. Finally, output the defect category, defect location and its boundary conditions.

[0138] Based on the method provided by this invention, this invention also provides a surface defect detection system for aluminum strip based on a lightweight model, comprising:

[0139] The raw image acquisition module is used to acquire raw image data of the aluminum strip surface in real time using an industrial line scan camera;

[0140] The image cropping and filtering module is used to evenly divide the original image data into multiple square sub-images and filter out the sub-images containing defects;

[0141] The image annotation module is used to perform pixel-level annotation on the selected defect sub-images, marking the defect contours and categories to obtain a surface defect dataset; the surface defect dataset includes a training set, a validation set, and a test set;

[0142] The data augmentation module is used to augment the surface defect dataset to obtain an augmented surface defect dataset.

[0143] The model building module is used to build a surface defect segmentation model, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network, and a prediction network.

[0144] The model training module is used to train, validate, and test the surface defect segmentation model using the data-enhanced surface defect dataset to obtain a trained surface defect segmentation model.

[0145] The surface defect detection module is used to detect surface defects in the aluminum strip production process in real time using the trained surface defect segmentation model.

[0146] Further, the present application also provides an electronic device, which can include a processor, a communication interface, a memory and a communication bus. Wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The processor can invoke the computer program in the memory to execute the aluminum strip surface defect detection method based on the lightweight model.

[0147] In addition, the computer program in the memory described above is realized in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0148] Further, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which can implement the aluminum strip surface defect detection method based on the lightweight model when executed.

[0149] The present application discloses an aluminum strip surface defect detection method, system and device based on a lightweight model, which relates to the field of industrial computers. The method mainly includes aluminum strip surface image acquisition, image cropping and screening, image labeling, pixel-level segmentation labeling of defects and assignment of class labels to obtain a surface defect dataset and divide it into a training set, a validation set and a test set, data enhancement, surface defect segmentation model construction, surface defect segmentation model training, saving of corresponding weight files, testing of each weight file on the validation set, selection of the round with the highest segmentation accuracy as the final model weight, and model generalization performance detection on the test set, surface defect segmentation prediction, input of the picture to be tested into the model and loading of the trained weight, and final output of the defect class, defect position and boundary condition. The present application greatly reduces the model parameter quantity and complexity while maintaining high segmentation accuracy, lays a theoretical foundation for the deployment of aluminum strip defect segmentation algorithms on mobile terminals and embedded devices, and has a wide application prospect.

[0150] The various embodiments described in this specification are presented for the purpose of illustrating the principles of the present application and its best mode of operation. Each of the embodiments described in this specification has been provided for the purpose of illustration only and the various embodiments are not intended to limit the present application in any way unless otherwise specifically indicated. The same parts and / or features of the various embodiments described in this specification can be referenced using the same reference numerals for the ease of understanding of the present application.

[0151] The principles and implementations of the present application have been described in the above embodiments, which are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation and application range of the present application can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for detecting surface defects in aluminum strip based on a lightweight model, characterized in that, include: Real-time acquisition of raw image data of aluminum strip surface using an industrial line scan camera; The original image data is evenly divided into multiple square sub-images, and the sub-images containing defects are filtered out. The selected defect sub-images are annotated at the pixel level to mark the defect contours and categories, resulting in a surface defect dataset; the surface defect dataset includes a training set, a validation set, and a test set; Data augmentation is performed on the surface defect dataset to obtain the augmented surface defect dataset. A surface defect segmentation model is constructed, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network, and a prediction network; The construction of the surface defect segmentation model specifically includes: constructing a joint loss function for the surface defect segmentation model; building a backbone feature extraction network based on the MobileViTv2 network and the CBAM attention module; building a multi-scale feature fusion network based on the SPPF module and the HRFPN network; establishing a deep supervision network based on convolutional layers and the joint loss function; constructing a prediction network based on convolutional layers; and connecting the backbone feature extraction network, the multi-scale feature fusion network, the deep supervision network, and the prediction network to form the surface defect segmentation model. The surface defect segmentation model is trained, validated, and tested using the data-augmented surface defect dataset to obtain a trained surface defect segmentation model. The trained surface defect segmentation model is used to detect surface defects in the aluminum strip production process in real time.

2. The method for detecting surface defects in aluminum strip based on a lightweight model according to claim 1, characterized in that, The step of performing pixel-level annotation on the selected defect sub-images, marking the defect contours and categories, and obtaining a surface defect dataset specifically includes: LabelMe is used to manually classify and pixel-level segment and label the defects in the selected defect sub-images, and different categories of defects are marked with different colors to form a JSON format label dataset. Convert the JSON format tag dataset into a PNG format tag dataset; For each type of defect sub-image, 20% of its corresponding PNG format image is randomly divided into the test set, 90% of the remaining images are divided into the training set, and the last 10% is the validation set. The test set, training set, and validation set, which consist of different categories of defect sub-images and their corresponding PNG format images, are integrated separately. The integrated datasets are then formatted in the same way as the VOC2007 dataset to obtain the surface defect dataset.

3. The method for detecting surface defects in aluminum strip based on a lightweight model according to claim 1, characterized in that, The process of data augmentation of the surface defect dataset to obtain an augmented surface defect dataset specifically includes: The surface defect dataset is augmented using one or more of the following data augmentation methods: random cropping, random horizontal flipping, random vertical flipping, scale jittering, color jittering, or Mosaic, resulting in an augmented surface defect dataset.

4. The method for detecting surface defects in aluminum strip based on a lightweight model according to claim 1, characterized in that, The process of training, validating, and testing the surface defect segmentation model using the data-augmented surface defect dataset to obtain a trained surface defect segmentation model specifically includes: The surface defect segmentation model was trained using the data-augmented training set, and the corresponding weight files were saved during the training process. Each weight file was tested on the data-augmented validation set and the hyperparameters of the surface defect segmentation model were adjusted. At the same time, the weight file with the highest segmentation accuracy was selected as the model parameters of the trained surface defect segmentation model. Input the augmented test set into the trained surface defect segmentation model to examine the model's approximate generalization performance.

5. A surface defect detection system for aluminum strip based on a lightweight model, characterized in that, include: The raw image acquisition module is used to acquire raw image data of the aluminum strip surface in real time using an industrial line scan camera; The image cropping and filtering module is used to evenly divide the original image data into multiple square sub-images and filter out the sub-images containing defects; The image annotation module is used to perform pixel-level annotation on the selected defect sub-images, marking the defect contours and categories to obtain a surface defect dataset; the surface defect dataset includes a training set, a validation set, and a test set; The data augmentation module is used to augment the surface defect dataset to obtain an augmented surface defect dataset. The model building module is used to build a surface defect segmentation model, including a backbone feature extraction network, a multi-scale feature fusion network, a deep supervision network, and a prediction network. The construction of the surface defect segmentation model specifically includes: constructing a joint loss function for the surface defect segmentation model; building a backbone feature extraction network based on the MobileViTv2 network and the CBAM attention module; building a multi-scale feature fusion network based on the SPPF module and the HRFPN network; establishing a deep supervision network based on convolutional layers and the joint loss function; constructing a prediction network based on convolutional layers; and connecting the backbone feature extraction network, the multi-scale feature fusion network, the deep supervision network, and the prediction network to form the surface defect segmentation model. The model training module is used to train, validate, and test the surface defect segmentation model using the data-enhanced surface defect dataset to obtain a trained surface defect segmentation model. The surface defect detection module is used to detect surface defects in the aluminum strip production process in real time using the trained surface defect segmentation model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the aluminum strip surface defect detection method based on the lightweight model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the aluminum strip surface defect detection method based on the lightweight model as described in any one of claims 1 to 4.

Images