Defect segmentation method, system and device for magnetic tile based on multi-scale feature fusion

Abstract

The application discloses a magnetic tile defect segmentation method, system and equipment based on multi-scale feature fusion, and relates to the field of industrial manufacturing. First, the backbone extraction network is replaced from Xception to PS-MobileNetXt, so that the overall parameter quantity of the model is reduced to meet the requirements of deployment in the actual industrial production environment. Then, a fixed multiple interval hole rate design idea is introduced, a parallel pooling operation with a hole rate of 3 is introduced in the hole space pyramid pooling module, the learning of local features is improved, and the feature expression ability for small target defects is improved. Finally, a pyramid space feature fusion module is introduced in the decoder module, different scale feature information is weighted and fused through pyramid grouping convolution combined with channel weight, the purpose of cross-region connection and model parameter quantity reduction is achieved. The application has good feature extraction effect for multi-scale magnetic tile defects, and performs well in evaluation indexes and visual results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial manufacturing, in particular to a magnetic shoe defect segmentation method, system and equipment based on multi-scale feature fusion. BACKGROUND

[0002] In recent years, China's traditional manufacturing industry has ushered in new challenges and opportunities, and product upgrading and transformation has ushered in great changes. The policy requirements for China's manufacturing industry have promoted the development of new energy electric vehicles, variable frequency home appliances, industrial robots and other fields, while putting forward high-level requirements for the quality, performance and safety of permanent magnet DC motors. As an important component of permanent magnet DC motors, the quality of ferrite magnetic shoes will further affect the working performance and service life of the motor. Due to the influence of internal and external factors such as impure raw materials, production environment and manufacturing process, the finished magnetic shoes will inevitably have defective products, which will cause the household appliances and military products using the motor to malfunction. Therefore, it is of great significance to segment the defects of the magnetic shoes.

[0003] In the past, small and medium-sized enterprises generally adopted manual detection for magnetic shoe surface defect detection. Although this method has good adaptability to complex defects, it is seriously coupled with the experience of detection personnel and the length of detection time, and ultimately leads to the occurrence of low and unstable detection accuracy.

[0004] With the breakthrough progress of deep learning technology in target detection, natural language processing and other computer vision tasks, it has gradually become a popular research field. Compared with the low efficiency and the problem of being greatly affected by the detection time of manual detection, the addition of deep learning method can reduce the investment of human resources, improve the detection efficiency, and realize the improvement of product production efficiency and the increase of economic benefits. On the one hand, due to the large difference in size and area of different categories of magnetic shoe defects, the existing deep learning method has poor extraction effect on the defect features of the magnetic shoes. Through segmentation, the defects are located to facilitate the judgment of the magnetic shoes in the later period whether to repair, cut or discard, but in this process, there will be small defect missing and large defect hole, which ultimately leads to low overall segmentation performance of the model. On the other hand, in order to improve the performance of the model, a large number of modules are introduced, which leads to the increase of the overall parameter quantity, so that the low-cost requirement of the industrial environment segmentation system deployment cannot be realized. SUMMARY

[0005] In view of the problems of low magnetic shoe defect segmentation accuracy and large model parameter quantity leading to the inability to realize the low-cost deployment requirement of the industrial environment in the background technology, the present application provides a magnetic shoe defect segmentation method, system and equipment based on multi-scale feature fusion, which can improve the magnetic shoe defect segmentation accuracy while reducing the parameter quantity, so as to realize the requirements of accurate segmentation and low-cost deployment in the industrial environment.

[0006] To achieve the above object, the present application provides the following scheme:

[0007] In one aspect, the present application provides a magnetic tile defect segmentation method based on multi-scale feature fusion, comprising:

[0008] The magnetic tile data set is subjected to class balancing processing, and training set, verification set and test set are divided;

[0009] The traditional semantic segmentation algorithm DeepLabv3+ is improved to obtain a PS-PSFFM DeepLabv3+ network;

[0010] The training set and the verification set samples are input into the PS-PSFFM DeepLabv3+ network for training to obtain a trained defect segmentation model, and the test set is used for testing;

[0011] The trained defect segmentation model is used to segment and predict the magnetic tile defects.

[0012] Optionally, the magnetic tile data set is subjected to class balancing processing, and training set, verification set and test set are divided, specifically comprising:

[0013] In view of the imbalance of each defect class in the magnetic tile data set, rotation, flipping and Gamma change are used to realize the class balancing processing of the magnetic tile data set, and the number of pictures of each class is expanded to a preset number;

[0014] In view of the inconsistency of the picture size of the magnetic tile data set, size normalization means is used to unify the image size;

[0015] The magnetic tile data set is divided into training set, verification set and test set according to the ratio of 7:2:1.

[0016] Optionally, the traditional semantic segmentation algorithm DeepLabv3+ is improved to obtain a PS-PSFFM DeepLabv3+ network, specifically comprising:

[0017] The PS-MobileNetXt network is used to replace the traditional Xception network at the encoder end of DeepLabv3+, a parallel pool with a hole rate of 3 is introduced into the Atrous Spatial Pyramid Pooling (ASPP) module to obtain an Improved Atrous Spatial Pyramid Pooling (IASPP) module, and a Pyramid Spatial Feature Fusion Module (PSFFM) is introduced at the parallel processing position of the 1 / 4 feature map at the decoder end to obtain a PS-PSFFM DeepLabv3+ network.

[0018] Optionally, the PS-MobileNetXt network comprises a MobileNetXt network pruned by level and a stripe pooling module.

[0019] The feature map is input into the MobileNetXt network pruned by level to obtain a feature map F mx .

[0020] The stripe pooling module is used to perform adaptive average pooling operations on the feature map F mx in width and height, respectively, to obtain two feature maps, which are then up-sampled and expanded to achieve the purpose of size consistency, and then the corresponding positions of the two feature maps are summed pixel by pixel to obtain a feature map F S ; and finally, the feature map F S is processed by a 1x1 convolution dimension increasing operation and a ReLU activation function to obtain a feature map F mx , which is multiplied by the pixel value of the corresponding position of the feature map F enc to obtain the final output feature map F .

[0021] Optionally, the input of the feature map into the MobileNetXt network pruned by level to obtain the feature map F mx specifically comprises:

[0022] The 320x352x1 feature map is first processed by a 3x3 two-dimensional convolution to increase the number of channels and halve the size of the feature map, obtaining a 160x176x32 feature map; then the 5 sandglass modules are processed in turn to obtain 80x88x96, 80x88x144, 40x44x192, 20x22x288, and 20x22x384 feature maps, and the 20x22x384 feature map is taken as the final output feature map F mx .

[0023] Optionally, the IASPP module comprises a 1*1 convolution layer, a 3*3 convolution layer with a hole rate of 3, a 3*3 convolution layer with a hole rate of 6, a 3*3 convolution layer with a hole rate of 12, a 3*3 convolution layer with a hole rate of 18, a global average pooling layer, a feature splicing layer and a 1*1 convolution layer connected in sequence.

[0024] The feature map F obtained through the PS-MobileNetXt network enc The IASPP module is input, and a feature map list F is obtained through 1*1 convolution, convolution with a hole rate of 3, convolution with a hole rate of 6, convolution with a hole rate of 12, convolution with a hole rate of 18 and global average pooling processing List I1 I2 I3 I4 I5 I6 Finally, the feature map F is obtained through splicing and convolution operation; wherein the feature maps F IASPP , F I1 , F I2 , F I3 , F I4 , F I5 and F I6 are outputs of a 1*1 convolution layer, a 3*3 convolution layer with a hole rate of 3, a 3*3 convolution layer with a hole rate of 6, a 3*3 convolution layer with a hole rate of 12, a 3*3 convolution layer with a hole rate of 18 and a global average pooling layer respectively.

[0025] Optionally, the PSFFM module comprises a 1*1 convolution layer, a 2-fold upsampling layer, a PSA module and a BN layer.

[0026] The 1 / 8 feature map F1 output by the PS-MobileNetXt network is processed by a 1*1 convolution to achieve the same number of channels as the 1 / 16 feature map F2, and a feature map F 1-1 is obtained.

[0027] The 1 / 16 feature map F2 is sequentially processed by a 2-fold upsampling layer, a PSA module and a BN layer to obtain a feature map F 2-1 .

[0028] The feature maps F 1-1 and F 2-1 are subjected to feature addition and ReLU activation operation to obtain a feature map F PSFFM .

[0029] On the other hand, the application also provides a magnetic tile defect segmentation system based on multi-scale feature fusion, comprising:

[0030] ​​​​​​A data set processing and division unit is configured to perform category equalization processing on the magnetic tile data set and divide the data set into a training set, a verification set and a test set.

[0031] A network model improvement unit is configured to improve a traditional semantic segmentation algorithm DeepLabv3+ to obtain a PS-PSFFM DeepLabv3+ network.

[0032] A network model training unit is configured to input the training set and the verification set samples into the PS-PSFFM DeepLabv3+ network for training, obtain a trained defect segmentation model, and test the trained defect segmentation model by using the test set.

[0033] A defect segmentation prediction unit is configured to use the trained defect segmentation model to perform segmentation prediction on the magnetic tile defects.

[0034] In another aspect, the application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the magnetic tile defect segmentation method based on multi-scale feature fusion when executing the computer program.

[0035] Optionally, the memory is a non-transitory computer readable storage medium.

[0036] According to the embodiments of the application, the following technical effects are achieved.

[0037] The application provides a magnetic tile defect segmentation method, system and device based on multi-scale feature fusion. The backbone extraction network is replaced from Xception to PS-MobileNetXt to reduce the overall parameter quantity of the model and meet the requirements of deployment in an industrial actual production environment. A fixed multiple interval hole rate design idea is introduced, a parallel pooling operation with a hole rate of 3 is introduced in an atrous spatial pyramid pooling (ASPP) module, the learning of local features is improved to improve the feature expression ability for small target defects. Finally, a pyramid spatial feature fusion module (PSFFM) is introduced in the decoder module. Through pyramid grouping convolution combined with channel weight, weighted fusion of different scale feature information is realized, and the purposes of cross-region connection and model parameter reduction are achieved. Experiments prove that the method has good feature extraction effect for multi-scale defects and performs well in evaluation indexes and visual results. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0039] Figure 1 A flow chart of a magnetic tile defect segmentation method based on multi-scale feature fusion of the present application;

[0040] Figure 2 A structural diagram of the PS-PSFFM DeepLabv3+ network of the present application;

[0041] Figure 3 A process diagram of feature maps through the stripe pooling module;

[0042] Figure 4 A schematic diagram of introducing an ASPP module with a cavity rate of 3 in the present application;

[0043] Figure 5 A structural diagram of the pyramid spatial feature fusion module (PSFFM) of the present application;

[0044] Figure 6 A structural diagram of the decoder in the PS-PSFFM DeepLabv3+ network of the present application;

[0045] Figure 7 A prediction result diagram of ablation experiments of each improved module in the PS-PSFFM DeepLabv3+ network of the present application;

[0046] Figure 8 A prediction result diagram of comparison experiments of the PS-PSFFM DeepLabv3+ network and other networks. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] The purpose of the present application is to provide a magnetic tile defect segmentation method, system and device based on multi-scale feature fusion, which can improve the accuracy of magnetic tile defect segmentation while reducing the parameter amount, to realize the requirements of accurate segmentation and low-cost deployment in industrial environment.

[0049] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easier to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Figure 1 A flow chart of a magnetic tile defect segmentation method based on multi-scale feature fusion according to the present application is shown in FIG. 1. Figure 1 A magnetic tile defect segmentation method based on multi-scale feature fusion, comprising:

[0051] Step 1: Perform class balancing processing on the magnetic tile dataset, and divide the training set, validation set and test set.

[0052] The magnetic tile dataset of the present application is the Magnetic Tile Surface Defect dataset, which has five defect categories, namely Blowhole, Break, Crack, Fray and Uneven, and the defect sample data of each category is different, and there is an unbalanced data class situation. In view of the above situation, rotation, flipping and Gamma transformation are used to realize data balancing processing. Further, since the picture sizes of the dataset are inconsistent, parallel computing cannot be realized by Pytorch DataLoader to improve the training speed. In view of the above situation, size normalization is used to obtain images with a width of 352 pixels and a height of 320 pixels. In addition, the label image in the dataset is composed of an array with pixel values between 0 and 255, which is not convenient for model training. In view of the above situation, the picture read into the model is changed into a matrix tensor (Tensor) with defects of 1 and background of 0. The processed dataset is divided into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0053] Therefore, the step 1 performs class balancing processing on the magnetic tile dataset, and divides the training set, validation set and test set, and specifically includes:

[0054] Step 1.1: In view of the unbalanced problem of each defect category in the magnetic tile dataset, rotation, flipping and Gamma transformation are used to realize class balancing processing of the magnetic tile dataset, and the number of pictures of each category is expanded to a preset number.

[0055] The magnetic tile data set adopts a Magnetic Tile Surface Defect DataSet, which is collected and disclosed by the Chinese Academy of Sciences. The data set has a total of 1334 pictures, of which 392 pictures are defect pictures, and the five types of defects, pores, cracks, damage, wear and delamination, have 115, 57, 85, 32 and 103 pictures respectively. The class imbalance problem is more serious. Taking the least sample class as the benchmark, the proportion of each class is: 3.59, 1.78, 2.66, 1, and 3.22. The smallest proportion is 1.78, which is enough to see that the data class imbalance problem is significant. Combined with the possible situations of the magnetic tile data set and its own characteristics, different expansion methods are used for different categories of defects by using flipping, rotation and Gamma change. Finally, the number of each defect category reaches 200, and the total number is 1000.

[0056] Different expansion methods are used for different categories of defect samples. The expansion method for each category is shown in Table 1. The header row of Table 1 represents the type of data augmentation, where Rotate_180 represents image rotation by 180 degrees, Top_bottom represents image flipping up and down, Left_right represents image flipping left and right, and Gamma_x represents image Gamma change processing with a gamma value of x. The header column represents the five defect types, and the hook in the cell indicates that the defect type uses the data augmentation method, and the horizontal line indicates that the method is not used.

[0057] Table 1 Expansion method for different categories of defect samples

[0058]

[0059] Step 1.2: To solve the problem of inconsistent image sizes in the magnetic tile data set, size normalization is used to unify the image size.

[0060] Because the magnetic tile defect data set has inconsistent picture sizes, parallel computing cannot be achieved through Pytorch's DataLoader to improve training speed. Therefore, according to the picture length and width statistics of the data set and previous research content, the image size is finally normalized to 352x320 pixels. The specific operation process is as follows: first, judge the value of width and height, require width greater than height; then transpose the picture whose height is greater than width to make the width greater than height; finally, through bilinear interpolation, all pictures and their corresponding labels are unified to 352 pixels in width and 320 pixels in height.

[0061] In addition, since the label image in the data set is composed of an array with pixel values between 0-255, it is not convenient for model training, therefore, the application aims at the above-mentioned situation, and changes the picture read into the model into a matrix tensor (Tensor) with defects of 1 and background of 0. By traversing each pixel point in the Tensor through a double for loop, the defect part (pixel value is 255) is re-assigned to 1 through an if-else conditional statement, and the background part (pixel value is not 255) is re-assigned to 0.

[0062] Step 1.3: The magnetic shoe data set is divided into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0063] Specifically, the data set is divided according to a ratio of 7:2:1, and the number of pictures in the training set, the validation set and the test set is 695, 197 and 43 respectively, wherein, except that the test set does not introduce enhanced data samples, the other two data sets all have enhanced samples, and the specific data set division quantity is shown in Table 2. Wherein, the Train, Val and Test in the title row respectively represent the training set, the validation set and the test set, the Source and Enhance respectively represent the original sample quantity and the enhanced sample quantity, and the title column, in addition to Total representing the sample quantity of the three data sets, the rest all represent the quantity of different properties of the magnetic shoe defect categories in different data sets.

[0064] Table 2 Data set division quantity

[0065]

[0066]

[0067] Step 2: The traditional semantic segmentation algorithm DeepLabv3+ is improved to obtain the PS-PSFFM DeepLabv3+ network.

[0068] The application improves the traditional semantic segmentation algorithm DeepLabv3+, replaces the traditional Xception network with the PS-MobileNetXt network at the encoder end of DeepLabv3+, introduces a parallel pool with a hole rate of 3 in the Atrous Spatial Pyramid Pooling (ASPP) module to obtain an improved Atrous Spatial Pyramid Pooling (IASPP) module, and introduces a pyramid spatial feature fusion module (PSFFM) at the parallel processing position of the 1 / 4 feature map at the decoder end to obtain the PS-PSFFM DeepLabv3+ network.

[0069] Specifically, the traditional semantic segmentation algorithm DeepLabv3+ model is divided into an encoder and a decoder, the encoder part is mainly composed of an Xception network and an atrous spatial pyramid pooling (ASPP) module, the former realizes sufficient extraction of feature information, and the latter improves the acquisition ability of the model to multi-scale targets. The decoder performs upsampling processing on the encoder output feature map, then fuses the low-level features obtained by the Xception network in this process, and finally obtains a segmentation mask of the original input image resolution size through bilinear interpolation to make up for the loss of precision of the bilinear interpolation of the encoder output feature map.

[0070] The present application makes further improvements on the basis of the above-mentioned classical segmentation DeepLabv3+ network, specifically including:

[0071] (1) The encoder in the DeepLabv3+ network is improved, and the PS-MobileNetXt network is used to replace the original backbone network Xception, and the specific network structure is shown in Table 3. Wherein, Num represents the serial number, Input represents the input picture size AxBxC (height x width x channel number), and the specific data comes from the high and wide of the magnetic tile data set; Output represents the output picture size; Operator represents the name of the current layer operation; t represents the attenuation factor of the channel number of each layer operation; s represents the step length of the convolution kernel movement; b represents the number of times of repeating the layer operation. The first 6 modules in Table 3 are the first six layers of the original MobileNetXt network, and the seventh module is the introduced stripe pooling module.

[0072] Table 3 PS-MobileNetXt network structure

[0073]

[0074] In the encoder part of the PS-PSFFM DeepLabv3+ network model, the PS-MobileNetXt network includes a MobileNetXt network subjected to hierarchical pruning and a stripe pooling module, that is, the feature map is subjected to hierarchical pruning and the MobileNetXt network and the stripe pooling module in sequence.

[0075] The lightweight magnetic tile defect segmentation network of the present application multi-scale feature fusion, namely the PS-PSFFM DeepLabv3+ network, uses a lightweight PS-MobileNetXt network to replace the Xception network as the backbone feature extraction network, inputs the feature map into the MobileNetXt network subjected to hierarchical pruning to obtain a feature map F mx , and then performs stripe pooling (StripPooling) processing to obtain a feature F enc .

[0076] Specifically, referring to Table 3 and Figure 2 , the process of the MobileNetXt network with hierarchical pruning of feature maps is as follows: the 320x352x1 feature map is first subjected to 3x3 two-dimensional convolution to increase the number of channels and halve the size of the feature map to obtain a 160x176x32 feature map; then the feature map is subjected to 5 sandglass (hourglass residual) modules to obtain 80x88x96, 80x88x144, 40x44x192, 20x22x288 and 20x22x384 feature maps in sequence, wherein the final output feature map F mx The sandglass module is composed of two deep separable convolutions at the beginning and end and two point-by-point convolutions in the middle, the former is used to explicitly extract spatial features, and the latter is used to expand and reduce the dimension, not only does the short connection at the high-dimensional feature layer save more feature information for back propagation, but also the introduction of the module improves the experimental effect, which proves that applying convolution to high-dimensional space helps to improve the learning representation ability of the model.

[0077] Referring to Figure 3 , the process of the feature map through the stripe pooling module is as follows: the feature map F mx is subjected to adaptive average pooling operations in width and height respectively to obtain a 20x1 feature map and a 1x22 feature map; then the two feature maps are expanded in size along the left-right and top-bottom directions respectively through convolution operations to obtain a 20x22 feature map; finally, the corresponding positions of the two feature maps are summed pixel by pixel to obtain a 20x22x96 pixel feature map F S . Finally, F S is subjected to dimension increasing operation by 1x1 convolution and ReLU activation function processing, and then multiplied by the pixel value of the corresponding position of F mx to obtain the final output feature map Fenc of 20x22x384. The stripe pooling module obtains feature information through a strip frame (Strip pool), improves the acquisition of context information with different shapes, and realizes the acquisition of information of non-square feature regions while avoiding the acquisition of too much useless information by a square frame.

[0078] (2) Improving the spatial pyramid pooling (ASPP) module in DeepLabv3+, in order to solve the problem that too large hole rate makes the receptive field too large, and then causes small defects in the magnetic tile sample to be ignored, the present application introduces a parallel pooling with a hole rate of 3 in the spatial pyramid pooling (ASPP) module, and builds an improved ASPP module (IASPP), the structure of which is shown in Figure 4 . The feature map F enc output by the feature extraction network is input into the IASPP module, and the feature maps output by multiple parallel pooling layers are spliced to obtain a feature map FIASPP Specifically, the feature map F enc is input into the IASPP module, and a feature map list F List is obtained through 1x1 convolution, convolution with a dilation rate of 3, convolution with a dilation rate of 6, convolution with a dilation rate of 12, convolution with a dilation rate of 18, and global average pooling processing I1 , F I2 , F I3 , F I4 , F I5 , F I6 , and finally the feature map F IASPP is obtained through splicing and convolution operation. The receptive field sizes of the dilation rates of 3-6-12-18 are 49, 169, 625, and 1369, respectively. Compared with the module without a dilation rate of 3, the receptive field size of 49 can better obtain the feature information of blowhole and crack with defect area sizes of 163 and 190, thereby making up for the problem that small targets are ignored due to large receptive fields of 625 and 1369. In the parallel pooling operation with a dilation rate of 6-12-18, adding a parallel pooling operation with a dilation rate of 3 can avoid the problem of small target loss caused by a large dilation rate, thereby effectively improving the feature extraction capability for small target defects. The receptive field calculation formula is RF i = RF i +{[Ker+(Ker-1)×(Dil-1)]-1}×S i ; where RF i represents the receptive field size of the i-th layer; Ker represents the convolution kernel size; Dil represents the dilation rate; and S i represents the product of the convolution kernel moving step sizes of all layers except the current layer.

[0079] (3) A pyramid spatial feature fusion module (PSFFM) is introduced into the decoder part of the DeepLabv3+ network, as shown in FIG. 3. Figure 5 The PSFFM is parallel to the original fusion encoder 1 / 4 feature map operation of the DeepLabv3+ network, and the 1 / 8 and 1 / 16 feature maps in the encoder are processed through the pyramid spatial feature fusion module (PSFFM). First, the F1 (1 / 8 feature map) in the feature extraction network is processed using a 1x1 convolution to achieve the same number of channels as the F2 (1 / 16 feature map), to obtain a feature map F 1-1 ; then the F2 is sequentially processed through 2x upsampling, a PSA module, and a BN layer normalization processing, to obtain a feature map F 2-1 ; finally, the feature maps F 1-1 and F 2-1 are subjected to feature addition and ReLU activation operation, to obtain a feature map F PSFFMThe processing procedure for the feature map F2 includes: firstly, bilinear interpolation operation is performed to realize the purpose of 2 times upsampling of the feature map, so as to achieve the purpose of the same receptive field size as the F1 feature map; secondly, the feature channels are grouped, and different convolutions are implemented for each group to obtain different sizes of receptive fields and feature information; then the weights extracted by the SE module are used to distinguish the importance of information between different channels; then the weights are normalized by Softmax and weighted calculation of each group of feature values is realized, and then the purpose of multi-region feature connection is realized through different sizes of context information; finally, the BN layer normalization processing is performed to accelerate the model training speed, and the feature map F 2-1 .

[0080] (4) The overall structure of the decoder in the PS-PSFFM DeepLabv3+ network is shown in Figure 6 , which is mainly used to realize the fusion of the encoder feature extraction network F0 (1 / 4 feature map), F1 (1 / 8 feature map), F2 (1 / 16 feature map) and the feature map F IASPP processed by the improved spatial pyramid module (IASPP). The feature map F1 and F2 are processed by the pyramid spatial feature fusion module to obtain the feature map F PSFFM ; the feature map F IASPP is 2 times up-sampled to obtain a 40x44x256 feature map and the feature map F PSFFM is added, and then 2 times up-sampling is performed to obtain an 80x88x256 feature map F3; the feature map F0 is processed by 1x1 dimension reduction convolution to obtain an 80x88x48 feature map, which is spliced with F3 to obtain an 80x88x256 feature map F4, and then 3x3 convolution smoothing operation and 4 times up-sampling operation are performed to obtain the final output of the model, a 320x352x1 feature map F out .

[0081] Step 3: input the training set and validation set samples into the PS-PSFFM DeepLabv3+ network for training, obtain the trained defect segmentation model, and test it by using the test set.

[0082] The training set is used to train the model, the validation set is used to evaluate the performance of the model, and the test set is used to evaluate the performance of the model on unknown data. In the process of model training, the model parameters are constantly adjusted to make the model perform best on the validation set. In the process of model testing, the test set is used to evaluate the performance of the model on unknown data, so as to evaluate whether the model meets the expected magnetic tile defect segmentation prediction effect. Specifically, for the hyperparameters required for network training, the batch size (BatchSize) is set to 4 and the iteration rounds (epochs) are set to 200. The model uses the Adam optimizer to set different learning rates for different parameters by calculating the first moment estimate and the second moment estimate, and after updating, the weight decay is used to improve the accuracy while preventing the model from overfitting. The betas parameter is set to (0.9, 0.999), and the eps parameter and the weight decay rate are both set to 1e -8 . The initial learning rate is set to 1e -3 , the update strategy uses the ExponentialLR function to adjust the learning rate exponentially, and the learning rate will decay exponentially. The initial learning rate is large, and as the training rounds increase, the learning rate will gradually decrease, avoiding the case that the learning rate is too large to cause the network not to converge and ignore the optimal solution. The loss function in training uses the binary cross-entropy loss function (BCE Loss) commonly used in binary classification.

[0083] Step 4: Use the trained defect segmentation model to segment and predict the magnetic tile defects.

[0084] Specifically, running the test code realizes the output of the prediction mask label (i.e. black and white binary image). On the one hand, the OpenCV official database call function is used to realize the loading of the image set of the magnetic tile test set; on the other hand, the trained model parameters are loaded through the model loading function torch.load to complete the construction of the test model.

[0085] Further, an ablation experiment and a comparison experiment are designed, and the evaluation indicators of semantic segmentation, such as segmentation accuracy Acc, mean intersection over union MIoU, defect Iou, and Dice coefficient, are used to verify the PS-PSFFM DeepLabv3+.

[0086] Specifically, on the basis of the model DeepLabv3+, the four evaluation indicators Acc, MIoU, Iou and Dice are first improved by 1.36%, 9.51%, 17.6% and 6.25% respectively after replacing Xception with PS-MobileNetXt network, then improved by 0.01%, 0.22%, 0.44% and 0.16% respectively after introducing parallel pooling with a hole rate of 3, and finally improved by 0.11%, 0.62%, 1.13% and 0.37% respectively after introducing the pyramid spatial feature fusion module (PSFFM). From the segmentation evaluation indicators, the improved components of the application have better improvement on the segmentation accuracy of the magnetic tile defects. The specific visualization results are shown in Figure 7 .

[0087] Specifically, the model PS-PSFFM DeepLabv3+ has values of 98.71%, 90.92%, 83.23% and 95.06% in the four evaluation indicators Acc, MIoU, Iou and Dice, respectively. Compared with the basic UNet, the indicators are 1.22%, 7.25%, 13.23% and 4.58% higher, respectively. Compared with the DeepLab series network, the highest improvement is compared with DeepLabv1, which is 1.15%, 7.98%, 14.76% and 5.09% higher, respectively. Although the smallest improvement is compared with DeepLabv2, the four evaluation indicators are also improved by 0.41%, 2.76%, 5.09% and 1.95%. From the segmentation evaluation indicators, the PS-PSFFM DeepLabv3+ network realized by the application has better improvement on the segmentation accuracy of the magnetic tile defects. The specific visualization results are shown in Figure 8 .

[0088] Specifically, in terms of the total number of model parameters, the parameter quantities of DeepLabv1, DeepLabv2, DeepLabv3, DeepLabv3+, UNet and the model PS-PSFFM DeepLabv3+ realized by the application are 23.50M, 23.58M, 33.94M, 54.70M, 13.43M and 6.49M, respectively. It can be seen that the PS-PSFFM DeepLabv3+ model has the least number of parameters, and compared with other models, it is reduced by 88.13% at most and 51.67% at least. The data proves that the PS-PSFFM DeepLabv3+ model effectively realizes the purpose of reducing the number of parameters by replacing the lightweight feature extraction network.

[0089] Based on the method provided by the application, the application also provides a magnetic tile defect segmentation system based on multi-scale feature fusion, comprising:

[0090] A data set processing and division unit is configured to perform category equalization processing on the magnetic tile data set and divide the training set, the validation set and the test set;

[0091] A network model improvement unit is configured to improve a traditional semantic segmentation algorithm DeepLabv3+ to obtain a PS-PSFFM DeepLabv3+ network.

[0092] A network model training unit is configured to input the training set and the validation set sample into the PS-PSFFM DeepLabv3+ network for training, obtain a trained defect segmentation model, and test the trained defect segmentation model by using the test set.

[0093] A defect segmentation prediction unit is configured to use the trained defect segmentation model to perform segmentation prediction on the magnetic tile defect.

[0094] The present application performs equalization processing on the magnetic tile data set category imbalance problem and divides the training set, the validation set and the test set, improves the traditional semantic segmentation algorithm DeepLabv3+ model, replaces the Xception network with PS-MobileNetXt at the encoder end and introduces a parallel pool with a hole rate of 3, and introduces a pyramid spatial feature fusion module at the decoder end to meet the requirements of model lightweight, reduce small defect omission rate and enhance feature fusion, inputs the magnetic tile defect sample into the improved model for training, obtains a trained magnetic tile defect segmentation model, and performs segmentation prediction on the trained model in the magnetic tile defect test sample set.

[0095] Further, the present application also provides an electronic device, which can include a processor, a communication interface, a memory and a communication bus. Wherein, the processor, the communication interface, the memory complete mutual communication through the communication bus. The processor can call the computer program in the memory to execute the magnetic tile defect segmentation method based on multi-scale feature fusion.

[0096] In addition, the computer program in the memory described above is realized in the form of a software function unit and sold or used as an independent product, and can be stored in a non-transitory computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0097] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0098] In the present application, specific examples are applied to illustrate the principles and implementation manners of the present application. The above embodiment is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A magnetic shoe defect segmentation method based on multi-scale feature fusion, characterized in that, The application relates to a magnetic tile defect segmentation method based on a PS-PSFFM DeepLabv3+ network. The application comprises the following steps: Class balancing is performed on a magnetic tile data set, and a training set, a verification set and a test set are divided; A traditional semantic segmentation algorithm DeepLabv3+ is improved to obtain a PS-PSFFM DeepLabv3+ network, specifically comprising the following steps: A PS-MobileNetXt network is used to replace a traditional Xception network at an encoder end of the DeepLabv3+; parallel pooling with a hole rate of 3 is introduced into an all-space pyramid pooling ASPP module to obtain an improved all-space pyramid pooling IASPP module; and a pyramid space feature fusion module PSFFM is introduced into a parallel processing position of a 1 / 4 feature map at a decoder end to obtain the PS-PSFFM DeepLabv3+ network; input the feature map into a MobileNetXt network pruned by level to obtain a feature map F mx ; The stripe pooling module is used for pooling the feature map F mx The width and height adaptive average pooling operations are respectively performed, two feature maps obtained are respectively up-sampled and expanded to achieve the purpose of consistent size, and then corresponding positions of the two are pixel-by-pixel summed to obtain a feature map F S ; finally, the feature map F S After the dimension increasing operation of 1*1 convolution and the ReLU activation function processing, the feature map F mx is multiplied with the feature map F enc corresponding position pixel values to obtain the final output feature map F The PS-MobileNetXt network comprises a MobileNetXt network subjected to hierarchical pruning and a stripe pooling module; feature map F obtained through the PS-MobileNetXt network enc An IASPP module is inputted, and a feature map list F List = {F I1 , F I2 , F I3 , F I4 , F I5 , F I6} is obtained through 1x1 convolution, convolution with a hole rate of 3, convolution with a hole rate of 6, convolution with a hole rate of 12, convolution with a hole rate of 18 and global average pooling processing, and finally a feature map F IASPP is obtained through splicing and convolution operation; wherein the feature maps F I1 , F I2 , F I3 , F I4 , F I5 and F I6 are outputs of a 1x1 convolution layer, a 3x3 convolution layer with a hole rate of 3, a 3x3 convolution layer with a hole rate of 6, a 3x3 convolution layer with a hole rate of 12, a 3x3 convolution layer with a hole rate of 18 and a global average pooling layer respectively. The IASPP module comprises a 1x1 convolution layer, a 3x3 convolution layer with a hole rate of 3, a 3x3 convolution layer with a hole rate of 6, a 3x3 convolution layer with a hole rate of 12, a 3x3 convolution layer with a hole rate of 18, a global average pooling layer, a feature splicing layer and a 1x1 convolution layer which are sequentially connected; The 1 / 8 feature map F1 output by the PS-MobileNetXt network is used for 1x1 convolution to achieve the same number of channels as the 1 / 16 feature map F2, and a feature map F 1-1 ; The 1 / 16 feature map F2 is sequentially subjected to a 2x upsampling layer, a PSA module, and a BN layer for normalization to obtain a feature map F 2-1 ; The feature map F 1-1 is obtained by performing feature addition and ReLU activation operation on F 2-1 and F PSFFM ; The PSFFM module comprises a 1x1 convolution layer, a 2-fold up-sampling layer, a PSA module and a BN layer; The training set and the verification set samples are input into the PS-PSFFM DeepLabv3+ network for training, a trained defect segmentation model is obtained, and the test set is used for testing; 2. The magnetic tile defect segmentation method based on multi-scale feature fusion according to claim 1, characterized in that, The trained defect segmentation model is used for magnetic tile defect segmentation and prediction. The application comprises the following steps: In view of the unbalanced problem of each defect category in the magnetic tile data set, rotation, flipping and Gamma change are used to realize class balancing of the magnetic tile data set, and the number of pictures of each category is expanded to a preset number; In view of the inconsistent problem of the picture size of the magnetic tile data set, size normalization means is used to unify the image size; 3. The magnetic tile defect segmentation method based on multi-scale feature fusion according to claim 1, characterized in that, The feature map is input into the MobileNetXt network pruned by the level to obtain a feature map F mx , and specifically comprises: The 320x352x1 feature map is first subjected to 3x3 two-dimensional convolution to increase the number of channels and halve the size of the feature map to obtain a 160x176x32 feature map; then the 160x176x32 feature map is subjected to five sandglass modules to obtain 80x88x96, 80x88x144, 40x44x192, 20x22x288, and 20x22x384 feature maps in sequence, and the 20x22x384 feature map is taken as the final output feature map F mx .

4. A magnetic shoe defect segmentation system based on multi-scale feature fusion, characterized in that, The magnetic tile data set is divided into the training set, the verification set and the test set according to a 7:2:1 ratio. The application comprises the following steps: A data set processing and division unit is used for performing class balancing on a magnetic tile data set and dividing a training set, a verification set and a test set; A network model improvement unit is used for improving a traditional semantic segmentation algorithm DeepLabv3+ to obtain a PS-PSFFM DeepLabv3+ network, specifically comprising the following steps: A PS-MobileNetXt network is used to replace a traditional Xception network at an encoder end of the DeepLabv3+; parallel pooling with a hole rate of 3 is introduced into an all-space pyramid pooling ASPP module to obtain an improved all-space pyramid pooling IASPP module; and a pyramid space feature fusion module PSFFM is introduced into a parallel processing position of a 1 / 4 feature map at a decoder end to obtain the PS-PSFFM DeepLabv3+ network; The PS-MobileNetXt network comprises a MobileNetXt network pruned through a hierarchical pruning and a stripe pooling module; inputting the feature map into a MobileNetXt network pruned by a level to obtain a feature map F mx ; The stripe pooling module is used for pooling the feature map F mx The width and height adaptive average pooling operations are respectively performed, two feature maps obtained are respectively up-sampled and expanded to achieve the purpose of consistent size, and then corresponding positions of the two are pixel-by-pixel summed to obtain a feature map F S ; finally, the feature map F S After the dimension increasing operation of 1*1 convolution and the ReLU activation function processing, the feature map F mx The pixel values of the corresponding positions are multiplied to obtain the final output feature map F enc ; The IASPP module comprises a 1x1 convolution layer, a 3x3 convolution layer with a hole rate of 3, a 3x3 convolution layer with a hole rate of 6, a 3x3 convolution layer with a hole rate of 12, a 3x3 convolution layer with a hole rate of 18, a global average pooling layer, a feature concatenation layer and a 1x1 convolution layer connected in sequence; feature map F obtained through the PS-MobileNetXt network enc An IASPP module is inputted, and a feature map list F List I1 I2 I3 I4 I5 I6 is obtained through 1×1 convolution, convolution with a hole rate of 3, convolution with a hole rate of 6, convolution with a hole rate of 12, convolution with a hole rate of 18 and global average pooling processing, and finally a feature map F IASPP is obtained through splicing and convolution operation; wherein the feature maps F I1 , F I2 , F I3 , F I4 , F I5 and F I6 are respectively outputs of a 1×1 convolution layer, a 3×3 convolution layer with a hole rate of 3, a 3×3 convolution layer with a hole rate of 6, a 3×3 convolution layer with a hole rate of 12, a 3×3 convolution layer with a hole rate of 18 and a global average pooling layer.​​​​​​ The PSFFM module comprises a 1x1 convolution layer, a 2x upsampling layer, a PSA module and a BN layer; The 1 / 8 feature map F1 output by the PS-MobileNetXt network is used for 1x1 convolution to achieve the same number of channels as the 1 / 16 feature map F2, and a feature map F 1-1 ; The 1 / 16 feature map F2 is sequentially subjected to a 2x upsampling layer, a PSA module, and a BN layer for normalization to obtain a feature map F 2-1 ; The feature map F 1-1 is obtained by performing feature addition and ReLU activation operation on F 2-1 and F PSFFM ; The network model training unit is configured to input training set and validation set samples into the PS-PSFFM DeepLabv3+ network for training, obtain a trained defect segmentation model, and test the trained defect segmentation model by using a test set. The defect segmentation prediction unit is configured to segment and predict the magnetic tile defects by using the trained defect segmentation model.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the magnetic tile defect segmentation method based on multi-scale feature fusion according to any one of claims 1 to 3 when executing the computer program.

6. The electronic device of claim 5, wherein, The memory is a non-transitory computer readable storage medium.

Images