A texture image defect classification system and method based on multi-resolution feature fusion

Abstract

The present invention discloses a system and method for high-precision texture image defect classification based on multi-resolution fusion, wherein the classification system comprises: an image acquisition module, a data processing module, a network training module and a defect classification module; the image acquisition module is used to acquire defect images; the data processing module is used to calibrate defect area location information, intercept defect samples, and establish a defect data set; the network training module uses a multi-resolution feature fusion network with a deep and shallow layer fusion structure to extract image features of each sample in the defect data set and perform defect classification network training; the defect classification module uses a trained defect classification network to extract image features of defect samples for defect classification. At the same time, the present invention also discloses a deep and shallow layer fusion structure for feature fusion, which can be applied to a variety of classification networks, so that features of different resolutions can be fused and participate in classification decisions.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a system and method for classifying defects generated in texture images. Background Art

[0002] my country has a large population and the textile industry, as a traditional labor-intensive industry, has always been a dominant industry in my country. It has made great contributions to my country's economic development and its textile output ranks first in the world for many years. However, with the continuous deepening of development, the textile industry has encountered problems that are inevitable in the development of labor-intensive industries, such as low added value, labor-intensive, lack of technical content and innovation, and low overall quality. In the weaving, dyeing, transportation and other links of the fabric, various defects appear on the surface of the fabric due to various reasons. The number of defects directly determines the quality of the fabric. Data show that the price of fabrics with more defects will be reduced by 45%-65%. Therefore, timely detection of defects and reduction of the number of defects are the key to improving fabric quality and price. The defects in the fabric are texture defects, which are characterized by being small and diverse, and the background texture itself is rich, which makes defect detection very difficult. From the perspective of the causes of formation and the characteristics of the defects, the defects are divided into three categories: warp defects, weft defects, and hole defects. The defect morphology can be seen in Figure 2 , where the first row is hole-shaped defects caused by rubbing holes, holes, and hair spots, the second row is warp defects caused by hanging warps, missing warps, and warp jumps, and the third row is weft defects caused by missing wefts and sparse weaving. From the shape of the defects, the hole-shaped defects are hole-shaped, the warp defects are strip-shaped defects consistent with the direction of the warp yarns of the loom, and the weft defects are strip-shaped defects consistent with the direction of the weft yarns of the loom. Early defect detection technology was based on machine learning for classification, relying on manual extraction of shallow features such as color, texture, and shape. It takes a lot of time to adjust, and the quality of the selection cannot be guaranteed. It is inevitable that redundant information and noise information will be introduced, making it impossible for the extracted features to objectively express the inherent attributes of the target, and these features cannot well represent the deep semantic information of the image. However, deep learning can automatically learn some features without manual participation in the feature selection process, and the deep features contain stronger high-level semantics, stronger resolution, and are more adaptable to classification tasks. In recent years, deep learning has achieved rapid progress in areas such as image recognition. It is possible to consider using deep learning-based defect detection technology to innovate the existing quality inspection process to reduce the incidence of missed inspections and improve product quality. Therefore, using deep learning technology to study texture defect classification and detection tasks is of great value to improving textile quality and production efficiency.

[0003] Deep learning is a process of automated feature extraction, such as Figure 6As shown in the figure, the shallow layer of the network extracts some low-level features of the edge. These shallow features (low-level features) have very rich detail information, which includes the edges and contours of objects and their combination with colors; the middle layer (middle-level features) begins to extract some local conceptual features, including some texture features; the deep layer of the network extracts some very high-level abstract features, and these deep features (high-level features) will reflect the differences between classes.

[0004] Deep features contain rich semantic information. Deep information contains more information and a large amount of data. Many existing studies have not taken shallow features into account, resulting in a huge amount of calculation, and the final classification accuracy and efficiency are not high. At present, there is a problem with the defect classification detection methods based on deep learning, such as mainstream networks such as ResNet and DenseNet, that is, only the output features of the last layer of the network are involved in the classification, and the shallow features are not directly involved in the final decision, resulting in low classification accuracy and efficiency. This is because in the mainstream deep learning network tasks, large targets pay more attention to the whole and have high semantic information, so deep features are often used as the input of the last fully connected layer. For example, Dong Amei (Research on Color Fabric Defect Detection and Classification Algorithm Based on Convolutional Neural Network [D]. Xi'an University of Technology, 2018.) accelerated the algorithm execution speed and improved the model accuracy by improving the data normalization method in the AlexNet network. Only the features of the last layer of the network were used, and the shallow features were not considered, resulting in a huge amount of calculation and low detection efficiency. In the defect classification task, we focus on information about the damage of defects to background textures, such as shallow features such as edges and gradients. The shallow features of neural networks reflect this information, so it is very necessary to reuse shallow features in deep features.

[0005] According to the previous analysis of defect characteristics and convolutional neural network feature extraction, the features at different levels of the network are fused, or the features at different resolutions are fused, so that these features can directly participate in classification decisions, which helps to improve the efficiency of defect classification. However, using the method of fusing features at different levels to detect defects requires solving two problems:

[0006] First, the feature resolutions at different levels are different, which means that features cannot be directly stacked or added together.

[0007] Secondly, the number of channels of features at different levels is also different. The number of channels of shallow features is small, while the number of channels of deep features is large. The number of channels of the fused features is too large, which will introduce a lot of calculations and reduce the model speed. Stacking or adding features in the channel dimension is the most commonly used method of feature fusion. Figure 7As shown, taking DenseNet121 as an example, in the DenseNet121 network structure, the size of the input image is 224×224×3, the scale of the features output by DenseBlock1 is 56×56×256, the scale of the features output by DenseBlock2 is 28×28×512, the scale of the features output by DenseBlock3 is 14x14x1024, and the scale of the features output by DenseBlock4 is 7×7×1024. It can be seen that the resolution of each DenseBlock output feature is different. In addition, if the features are stacked in the channel dimension, the dimension of the fused feature channel will reach 2816. ResNet, VGG, Inception and other networks have the same problem. For example, the multi-resolution feature fusion method proposed in the invention (publication number CN111784644A, a method and system for detecting printing defects based on deep learning) uses bilinear interpolation and convolution to expand feature maps of different resolutions to the same size for fusion, resulting in too many feature channels after fusion, large amount of calculation and slow calculation speed. The existing technology cannot directly fuse deep features and shallow features with small computational complexity and fast computational speed.

[0008] The existing technologies all have the above problems. Deep features and shallow features cannot be directly stacked or added. When deep features and shallow features are stacked and added in the channel dimension by means of expansion, a large amount of calculation is inevitably introduced, so shallow features cannot be efficiently introduced into the field of defect detection. Summary of the invention

[0009] In order to overcome the technical bias that the existing technology cannot directly fuse deep features and shallow features, and that the fusion of deep features and shallow features using the existing technology introduces a very large amount of calculation, the present invention provides a texture image defect classification system based on multi-resolution feature fusion. The present invention introduces a multi-resolution feature fusion network with a deep and shallow fusion structure, extracts shallow features such as edges of texture images and deep features. The present invention only needs to use a convolution and pooling neural network to reduce features of different resolutions and different levels to the same size, and can directly fuse them. At the same time, the amount of calculation is smaller and the calculation speed is faster. In addition, the present invention utilizes the shape adaptability of deformable convolution, and the classification accuracy is higher. In the defect classification task, the focus is on information such as the damage of defects to the background texture, such as shallow features such as edges and gradients, and the shallow features of the neural network reflect this information. It is very necessary to reuse shallow features in deep features.

[0010] The present invention discloses a texture image defect classification system based on multi-resolution feature fusion, wherein the defect classification system directly fuses shallow features and deep features of defects with different resolutions to classify the defects; the shallow features are bottom-layer features of edges, including edges and contours of objects and combinations with colors; the deep features are high-level abstract features; the defect classification system comprises: an image acquisition module, a data processing module, a defect classification network training module and a defect classification module;

[0011] The image acquisition module is used to acquire defect images;

[0012] The data processing module is used to calibrate the position information of the defect area, intercept the defect samples, input the defect samples into a common feature extraction network to obtain features of different resolutions, and establish a feature data set, wherein the features include deep features and shallow features;

[0013] The defect classification network training module uses a deep and shallow multi-resolution feature fusion network to extract the features of each sample in the feature data set and perform defect classification network training; the deep and shallow multi-resolution feature fusion network consists of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature so that the feature sizes of the deep layer and the shallow layer output are consistent, and a stacking operation can be performed directly in the channel dimension; preferably, the average pooling layer reduces the size of the feature by calculating the average value within the pooling window;

[0014] The defect classification module uses the trained defect classification network to extract defect features for classifying defects;

[0015] Preferably, the commonly used feature extraction network is ResNet, DenseNet, or VGGNet network.

[0016] Preferably, the deformable convolution layer is composed of two parallel convolution layers that share feature inputs; one of the two layers uses a conventional convolution layer, wherein the convolution kernel size used by the conventional convolution layer is 1 and the stride is 1; the other layer uses an additional convolution layer to learn position offsets in the x and y directions, and the number of output channels is twice that of the output of the conventional convolution layer.

[0017] Preferably, the dimensions of the features of different resolutions are h×j×c, where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature, and h, j, and c of deep features and shallow features may be different;

[0018] After the features of different resolutions are input into the regular convolution layer in the deformable convolution layer, the feature dimension of the first regular output feature is Where d represents the ratio of compressing the input channel;

[0019] After the features of different resolutions are input into the additional convolution layer in the deformable convolution layer, the feature dimension of the first additional output feature is

[0020] The first regular output feature and the first additional output feature are combined to obtain the second output feature, whose feature dimension is

[0021] The second output feature map is input into an average pooling layer to compress the size of the second output feature map, and the feature dimension of the third output feature map is obtained as Where k represents the size of the pooling window; different k are used for deep features and shallow features. and The same means that the feature sizes of the deep features and the shallow features output are consistent, and stacking operations can be performed directly in the channel dimension.

[0022] Preferably, a deformable convolution is performed on the feature r, and the output is:

[0023]

[0024] Among them, R represents the feature space of the input features, p n is an enumeration of all positions in the feature space, p0 is the current pixel position, Δp n represents the position offset, W(·) represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature, and f(·) corresponds to the pixel value at the specified position of the output feature;

[0025] Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows:

[0026]

[0027] Where p = p0 + p n +Δp n represents the sample position after the shift, represents the position after the shift, G(·,·) is the bilinear interpolation weight, where

[0028] G(a, b) = max(0, 1-|a x -b x |)·max(0,1-|a y -b y |)

[0029] Where (a x , a y ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, (bx , b y ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

[0030] Preferably, the method for intercepting defect samples is to count the length of the short side of the defect area, intercept the defect area as the defect sample according to the result, and intercept the non-defect area with the same size as the normal sample;

[0031] Preferably, the normal samples and defective samples are taken from the same picture to ensure that the defective samples are consistent with the normal samples in other factors except the presence or absence of defects, such as lighting and shooting angle.

[0032] Preferably, the method for establishing the data set is to manually label the defect samples and normal samples to establish four types of label data: warp defects, weft defects, hole defects or no defects.

[0033] Preferably, the calibrating the defect area position information includes calibrating the coordinates of the upper left corner and the lower right corner of the defect area.

[0034] The present invention also discloses a texture image defect classification method based on multi-resolution feature fusion, wherein the defect classification method directly fuses shallow features and deep features of defects with different resolutions, and classifies the defects after fusion. The defect classification method comprises the following steps:

[0035] Step (a-1), using an image acquisition module to collect images of defects to be tested;

[0036] Step (a-2), using a data processing module to calibrate the defect area position information of the defect image to be tested, intercepting the defect sample to be tested, inputting the defect sample into a commonly used feature extraction network to obtain features of different resolutions, and establishing a feature data set, wherein the features include deep features and shallow features;

[0037] Step (a-3), using the deep and shallow multi-resolution feature fusion network in the defect classification network training module to extract the features of each sample in the defect data set, and perform defect classification network training; the deep and shallow multi-resolution feature fusion network consists of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature, so that the feature size of the deep layer and the shallow layer output is consistent, and a stacking operation can be performed directly in the channel dimension; preferably, the average pooling layer reduces the size of the feature by calculating the average value within the pooling window;

[0038] Step (a-4), using a defect classification module to classify defects;

[0039] Preferably, the commonly used feature extraction network is ResNet, DenseNet, or VGGNet network.

[0040] Preferably, the feature dimensions of the features of different resolutions are h×j×c, where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature, and h, j, and c of deep features and shallow features may be different;

[0041] Preferably, the deep and shallow multi-resolution feature fusion network is implemented in the following steps:

[0042] Step (a-31) inputs the features of different resolutions into the regular convolution layer in the deformable convolution layer to obtain the first regular output feature, whose feature dimension is Where d represents the ratio of compressing the input channel;

[0043] Step (a-32) inputs the features of different resolutions into an additional convolution layer in the deformable convolution layer, and obtains the feature dimension of the first additional output feature as Where d represents the ratio of compressing the input channel;

[0044] Step (a-33) combines the first conventional output feature and the first additional output feature to obtain a second output feature, whose feature dimension is

[0045] Step (a-34) inputs the second output feature map into an average pooling layer, compresses the size of the second output feature map, and obtains a third output feature map with a feature dimension of Where k represents the size of the pooling window; different k are used for deep features and shallow features, so that the deep features and shallow features are and The same means that the feature sizes of the deep features and the shallow features output are consistent, and stacking operations can be performed directly in the channel dimension.

[0046] Among them, step (a-31) and step (a-32) have no particular order and can be performed sequentially or in parallel.

[0047] Preferably, the method for obtaining the defect sample to be tested in step (a-2) comprises the following steps:

[0048] Step (b-1), calibrate the defect image to be tested collected in step (a-1), and convert the defect area position information from (x min , x max ,y min ,y max ) four values ​​are calibrated, where (x min ,y min ) represents the coordinate of the upper left corner of the defect area, (xmax ,y max ) represents the coordinates of the lower right corner of the defect area;

[0049] Step (b-2), intercept the defect sample to be tested: min(x max -x min ,y max -y min ) Calculate the short side of the defect area, count the length of the short side, and select the appropriate image size based on the result;

[0050] Step (b-3), using a frame of a selected size to intercept the defect area and the normal area, and obtaining a defect sample to be tested and a normal sample respectively.

[0051] Preferably, the deformable convolution layer is composed of two parallel convolution layers, sharing feature input;

[0052] One of them uses a conventional convolutional layer with a kernel size of 1 and a stride of 1;

[0053] The other path uses an additional convolutional layer to learn the position offset in the x and y directions, and the number of output channels is twice that of the regular convolutional layer output;

[0054] Perform a deformable convolution on the feature map r, and the output is:

[0055]

[0056] Among them, R represents the feature space of the input feature map, p n is an enumeration of all positions in the feature space, p0 is the current pixel position, Δp n represents the position offset, W(·) represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature map, and f(·) represents the pixel value at the specified position of the output feature map;

[0057] Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows:

[0058]

[0059] Where p = p0 + p n +Δp n , represents the position after offset, G(·,·) is the bilinear interpolation weight;

[0060] G(a, b) = max(0, 1-|a x -b x |)·max(0,1-|a y -b y |)

[0061] Where (a x , a y ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, (b x , b y ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

[0062] Beneficial effects:

[0063] 1. Aiming at the particularity of defects generated in texture images, the present invention designs a special multi-resolution feature fusion texture image defect classification system, which fuses features at different levels, or in other words, fuses features at different resolutions, so that these features can directly participate in defect classification decisions, thereby improving the efficiency of defect classification;

[0064] 2. The present invention cleverly designs a special network feature extraction method for multi-resolution feature fusion, uses a deformable convolution layer to compress the number of channels of the input features, so that the number of channels of features at different levels decreases when stacked; uses an average pooling layer to reduce the size of the feature map, so that the feature maps at different levels are of the same size and can be directly stacked in the channel dimension; and realizes the stacking of feature maps at different levels through only one convolution layer and pooling layer, and greatly reduces the number of channels, greatly improving the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Get method for defect dataset.

[0066] Figure 2 Different types of defect images.

[0067] Figure 3 It is a deep and shallow multi-resolution feature fusion network.

[0068] Figure 4 This paper presents a texture image defect classification method based on multi-resolution feature fusion, taking DenseNet121 as an example.

[0069] Figure 5 , is a texture image defect classification system based on multi-resolution feature fusion.

[0070] Figure 6 , visualization of convolutional neural network feature extraction.

[0071] Figure 7 、DenseNet121 network feature size diagram DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below by taking cloth defects as an example in combination with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] like Figure 5 The present invention discloses a texture image defect classification system based on multi-resolution feature fusion, wherein the defect classification system directly fuses shallow features and deep features of defects with different resolutions to classify the defects; the shallow features are bottom-layer features of edges, including edges and contours of objects and combinations with colors; the deep features are high-level abstract features; the defect classification system comprises: an image acquisition module, a data processing module, a defect classification network training module and a defect classification module;

[0074] The image acquisition module is used to acquire defect images;

[0075] The data processing module is used to calibrate the position information of the defect area, intercept the defect samples, input the defect samples into a common feature extraction network to obtain features of different resolutions, and establish a feature data set, wherein the features include deep features and shallow features;

[0076] The defect classification network training module uses a deep and shallow multi-resolution feature fusion network to extract the features of each sample in the feature data set and perform defect classification network training; the deep and shallow multi-resolution feature fusion network consists of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature so that the feature sizes of the deep layer and the shallow layer output are consistent, and a stacking operation can be performed directly in the channel dimension; preferably, the average pooling layer reduces the size of the feature by calculating the average value within the pooling window;

[0077] The defect classification module uses the trained defect classification network to extract defect features for classifying defects;

[0078] Preferably, the commonly used feature extraction network is ResNet, DenseNet, or VGGNet network.

[0079] Preferably, the deformable convolution layer is composed of two parallel convolution layers that share feature inputs; one of the two layers uses a conventional convolution layer, wherein the convolution kernel size used by the conventional convolution layer is 1 and the stride is 1; the other layer uses an additional convolution layer to learn position offsets in the x and y directions, and the number of output channels is twice that of the output of the conventional convolution layer.

[0080] Preferably, the dimensions of the features of different resolutions are h×j×c, where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature, and h, j, and c of deep features and shallow features may be different;

[0081] After the features of different resolutions are input into the regular convolution layer in the deformable convolution layer, the feature dimension of the first regular output feature is Where d represents the ratio of compressing the input channel;

[0082] After the features of different resolutions are input into the additional convolution layer in the deformable convolution layer, the feature dimension of the first additional output feature is

[0083] The first regular output feature and the first additional output feature are combined to obtain the second output feature, whose feature dimension is

[0084] The second output feature map is input into an average pooling layer to compress the size of the second output feature map, and the feature dimension of the third output feature map is obtained as Where k represents the size of the pooling window; different k are used for deep features and shallow features, so that the deep features and shallow features are and The same means that the feature sizes of the deep features and the shallow features output are consistent, and stacking operations can be performed directly in the channel dimension.

[0085] Preferably, a deformable convolution is performed on the feature r, and the output is:

[0086]

[0087] Among them, R represents the feature space of the input features, p n is an enumeration of all positions in the feature space, p0 is the current pixel position, Δp n represents the position offset, W(·) represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature, and f(·) corresponds to the pixel value at the specified position of the output feature;

[0088] Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows:

[0089]

[0090] Where p = p0 + p n +Δp n represents the sample position after the shift, represents the position after the shift, G(·,·) is the bilinear interpolation weight, where

[0091] G(a, b) = max(0, 1-|a x -b x |)·max(0,1-|a y -b y |)

[0092] Where (a x , a y ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, (b x , b y ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

[0093] The average pooling layer reduces the size of the feature map by calculating the average value within the pooling window, so that the feature maps output by different blocks are of the same size, so that they can be stacked in the channel dimension. Pooling is to convert the pixel values ​​in a specified size area in the image into one value by averaging or directly taking the maximum value, so as to reduce the size of the image. The pooling window is the area of ​​this specified size. For example, a 4*4 image, set the pooling window to 2*2, and the step size to 2, and it will become a 2*2 image through pooling.

[0094] Preferably, the method for intercepting defect samples is to count the length of the short side of the defect area, intercept the defect area as the defect sample according to the result, and intercept the non-defect area with the same size as the normal sample;

[0095] Preferably, the normal samples and defective samples are taken from the same picture to ensure that the defective samples are consistent with the normal samples in other factors except the presence or absence of defects, such as lighting and shooting angle.

[0096] Preferably, the method for establishing the data set is to manually label the defect samples and normal samples to establish four types of label data: warp defects, weft defects, hole defects or no defects. From the perspective of the causes of defects and the characteristics of defects, the defects in the weaving stage are divided into three categories: warp defects, weft defects and hole defects. The defect morphology is shown in Figure 2 The first row is hole-shaped defects caused by rubbing holes, tears, and hair spots; the second row is warp defects caused by hanging warps, missing warps, and warp skipping; the third row is weft defects caused by missing weft and thin weaving.

[0097] Preferably, the calibrating the defect area position information includes calibrating the coordinates of the upper left corner and the lower right corner of the defect area.

[0098] The present invention also discloses a texture image defect classification method based on multi-resolution feature fusion, wherein the defect classification method directly fuses shallow features and deep features of defects with different resolutions, and classifies the defects after fusion. The defect classification method comprises the following steps:

[0099] Step (a-1), using the image acquisition module to collect the defect image to be tested; the image size is selected to be 224×224, which is more suitable, and the normal sample and the defect sample are selected from the same cloth image. Figure 1 The figure shows the method of obtaining the data set. In the figure, box 1 is the area where the defect is located in the original image. The method of obtaining the defective cloth sample is as follows: the defective area is intercepted with a 224×224 frame, and the overlapping area between the frames is randomly selected from 0 to 32 pixels. The position of the defect in the frame is random, and the defective image can be obtained, such as Figure 1 As shown in the middle boxes 2 to 4, randomly select a 224x224 image of the non-defective area in the image as a normal sample, such as Figure 1 As shown in the middle boxes 5 to 7. The images contained in this data set are non-printed images, which contain various defects generated during the weaving process, including rubbing holes, hanging warps, hair spots, missing warps, skipped flowers, and sparse weaving. A total of 1168 defective images and 1316 normal images were collected, with a resolution of 2560x1920. After collecting defects, training sets and validation sets were obtained by stratified random sampling according to categories, with 85% of the training set and 15% of the validation set.

[0100] Step (a-2), using a data processing module to calibrate the defect area position information of the defect image to be tested, intercepting the defect sample to be tested, inputting the defect sample into a commonly used feature extraction network to obtain features of different resolutions, and establishing a feature data set, wherein the features include deep features and shallow features;

[0101] Step (a-3), using the deep and shallow multi-resolution feature fusion network in the defect classification network training module to extract the features of each sample in the defect data set, and perform defect classification network training; the deep and shallow multi-resolution feature fusion network consists of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature, so that the feature size of the deep layer and the shallow layer output is consistent, and a stacking operation can be performed directly in the channel dimension; preferably, the average pooling layer reduces the size of the feature by calculating the average value within the pooling window;

[0102] Step (a-4), using a defect classification module to classify defects;

[0103] Preferably, the commonly used feature extraction network is ResNet, DenseNet, or VGGNet network.

[0104] Preferably, the feature dimensions of the features of different resolutions are h×j×c, where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature, and h, j, and c of deep features and shallow features may be different;

[0105] As a preference, Figure 3 As shown, the implementation steps of the deep and shallow multi-resolution feature fusion network are as follows:

[0106] Step (a-31) inputs the features of different resolutions into the regular convolution layer in the deformable convolution layer to obtain the first regular output feature, whose feature dimension is Where d represents the ratio of compressing the input channel;

[0107] Step (a-32) inputs the features of different resolutions into an additional convolution layer in the deformable convolution layer, and obtains the feature dimension of the first additional output feature as Where d represents the ratio of compressing the input channel;

[0108] Step (a-33) combines the first conventional output feature and the first additional output feature to obtain a second output feature, whose feature dimension is

[0109] Step (a-34) inputs the second output feature map into an average pooling layer, compresses the size of the second output feature map, and obtains a third output feature map with a feature dimension of Where k represents the size of the pooling window; different k are used for deep features and shallow features, so that the deep features and shallow features are and The same means that the feature sizes of the deep features and the shallow features output are consistent, and stacking operations can be performed directly in the channel dimension.

[0110] Step (a-31) and step (a-32) can be performed in any order and can be performed in sequence or in parallel. Figure 3 The k in AvgPool2d represents the size of the pooling window. The role of the pooling layer is to reduce the size of the feature map so that the feature maps output by different DenseBlocks are of the same size so that they can be stacked in the channel dimension.

[0111] Preferably, the method for obtaining the defect sample to be tested in step (a-2) comprises the following steps:

[0112] Step (b-1), calibrate the defect image to be tested collected in step (a-1), and convert the defect area position information from (x min , x max ,y min ,y max) four values ​​are calibrated, where (x min ,y min ) represents the coordinate of the upper left corner of the defect area, (x max ,y max ) represents the coordinates of the lower right corner of the defect area;

[0113] Step (b-2), intercept the defect sample to be tested: min(x max -x min ,y max -y min ) Calculate the short side of the defect area, count the length of the short side, and select the appropriate image size based on the result;

[0114] Step (b-3), using a frame of a selected size to intercept the defect area and the normal area, and obtaining a defect sample to be tested and a normal sample respectively.

[0115] Preferably, the deformable convolution layer is composed of two parallel convolution layers, sharing feature input;

[0116] One of them uses a conventional convolutional layer with a kernel size of 1 and a stride of 1;

[0117] The other path uses an additional convolutional layer to learn the position offset in the x and y directions, and the number of output channels is twice that of the regular convolutional layer output;

[0118] Perform a deformable convolution on the feature map r, and the output is:

[0119]

[0120] Among them, R represents the feature space of the input feature map, p n is an enumeration of all positions in the feature space, p0 is the current pixel position, Δp n represents the position offset, W(·) represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature map, and f(·) represents the pixel value at the specified position of the output feature map;

[0121] Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows:

[0122]

[0123] Where p = p0 + p n +Δp n , represents the offset position, G(·,·) is the bilinear interpolation weight, where

[0124] G(a, b) = max(0, 1-|a x -bx |)·max(0,1-|a y -b y |)

[0125] Where (a x , a y ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, (b x , b y ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

[0126] The average pooling layer reduces the size of the feature map by calculating the average value within the pooling window, so that the feature maps output by different blocks are of the same size, so that they can be stacked in the channel dimension. Specific implementation method:

[0128] Taking the DenseNet121 network as an example, the texture image defect classification method based on multi-resolution feature fusion after introducing deep and shallow layer fusion is as follows Figure 4 As shown, the specific parameters of the network are designed as follows:

[0129] 1) In this network, the input image size is 224×224×3, the scale of feature f1 output by DenseBlock1 is 56×56×256, the scale of feature f2 output by DenseBlock2 is 28×28×512, the scale of feature f3 output by DenseBlock3 is 14×14×1024, and the scale of feature f4 output by DenseBlock4 is 7×7×1024.

[0130] 2) In the deep and shallow multi-resolution feature fusion network modules 1, 2, and 3, the input channel compression ratio in the convolution layer is set to 2, the pooling window size in the pooling layer is set to 8, 4, and 2, respectively, and the pooling stride is equal to the pooling window size. After f1, f2, and f3 pass through the deep and shallow multi-resolution feature fusion network, the sizes of f1', f2', and f3' are 7×7×128, 7×7×256, and 7×7×512, respectively.

[0131] 3) f1', f2', f3' and f4 are stacked on the channel to obtain f4' of size 7x7x1920. f4' then passes through a convolution layer with a convolution kernel size of 1x1 and an output of 1024 dimensions to obtain a feature of size 7x7x1024, which is used as the feature for the final classification.

[0132] The following evaluation indicators are used for the texture image defect classification method based on multi-resolution feature fusion:

[0133] 1) Classification accuracy: Classification accuracy is defined as the ratio of the number of correctly classified samples to the total number of samples.

[0134] 2) Precision and Recall:

[0135]

[0136]

[0137] Among them, TP represents the number of positive classes predicted by the model as positive classes, FP represents the number of negative classes predicted by the model as positive classes, and FN represents the number of positive classes predicted by the model as negative classes.

[0138] Precision is also called the precision rate, which indicates the ratio of the number of correct defects detected by the model to the number of all defects detected by the model; recall is also called the recall rate, which indicates the ratio of the number of correct defects detected by the model to the number of all defects in the test set. The higher the precision rate, the fewer false positives the algorithm has; the higher the recall rate, the fewer missed positives the algorithm has.

[0139] 3) ROC curve and AUC

[0140] The final output of the deep learning classification model is a probability prediction. After obtaining the probability value, a threshold needs to be set to judge the prediction result. Different thresholds may result in different accuracy. The setting of the threshold will affect the recall and precision. When the threshold is set higher, the precision will be high, and vice versa. Sort the probability predictions of the samples, and use the probability predictions of the samples one by one from high to low as the threshold. Samples above the threshold are judged as positive samples, and vice versa. For each different threshold, the true positive rate and the false positive rate are calculated. The curve drawn with the false positive rate as the horizontal axis and the true positive rate as the vertical axis is called the ROC (Receiver Operating Characteristic) curve, that is, the "receiver operating characteristic" curve. The ROC curve describes the quality of a classifier from a global perspective.

[0141] AUC (Area Under ROC Curve) refers to the area under the ROC curve. The larger the AUC, the better the performance of the classifier. AUC is a binary classification indicator. In defect detection, the model's ability to detect defects is often more important than the model's ability to determine what type of defect it is. Therefore, in the classification task, we will treat samples with defects as one class and normal samples as another class in the validation set, and calculate the AUC indicator to measure the model's ability to detect defects.

[0142] The comparison of experimental results before and after multi-resolution feature fusion on the woven fabric dataset is shown in Table 1. It can be seen from the table that the network MRFF-DenseNet121 (DenseNet comes from the paper "Densely Connected Convolutional Networks") after fusing multi-resolution features has an accuracy of 87.75% on the woven fabric defect dataset, which is 4.11% higher than DenseNet121; MRFF-ResNet50 (ResNet comes from the paper "Deep Residual Learning for Image Recognition". The suffix numbers indicate the number of their network layers, and the prefix MRFF indicates the network structure after the fusion of deep and shallow layers.) has an accuracy of 87.22% on the woven fabric defect dataset, which is 1.63% higher than ResNet50. At the same time, MRFF-DenseNet121 has an accuracy improvement of 4.47% and a recall improvement of 5.2% compared with DenseNet121. MRFF-ResNet50 has an accuracy improvement of 4.43% and a recall improvement of 1.09% compared with ResNet50.

[0143] Table 1. Comparison of experimental results before and after multi-resolution feature fusion on woven fabric dataset

[0144]

[0145] The present invention has been described in detail above. The above specific implementation modes or embodiments are merely examples to reveal the technical contents of the present invention. The present invention is not limited to the above specific examples. The present invention should not be interpreted narrowly, but can be implemented with various changes within the spirit of the present invention and the scope of the claims.

[0146] In the above implementation, the deep and shallow layer fusion structure is applied to DenseNet121, and can also be used in various classification network structures such as ResNet.

[0147] Taking the same defect image as an example, the input image size resolution is 224××224, ResNet50 is used as the basic model, the parameters are optimized using the SGD algorithm, and the learning rate is set to 0.01. The cross entropy loss function is used. The calculation amount (Mult-Adds, multiplication and addition operations) of the present invention is 4806M, and the calculation amount (Mult-Adds, multiplication and addition operations) of the printing defect detection method and system based on deep learning in publication number CN111784644A is 12261M, so the overall calculation amount is about 3 times that of the present invention, which greatly reduces the calculation amount.

Claims

1. A texture image defect classification system based on multi-resolution feature fusion, characterized in that: The defect classification system directly fuses the shallow features and deep features of defects with different resolutions to classify the defects; the shallow features are the underlying features of the edge, including the edge and contour of the object and the combination with the color; the deep features are high-level abstract features; the defect classification system includes: an image acquisition module, a data processing module, a defect classification network training module and a defect classification module; The image acquisition module is used to acquire defect images; The data processing module is used to calibrate the position information of the defect area, intercept the defect samples, input the defect samples into a common feature extraction network to obtain features of different resolutions, and establish a feature data set, wherein the features include deep features and shallow features; The defect classification network training module uses a deep and shallow multi-resolution feature fusion network to extract features of each sample in the feature data set and perform defect classification network training; the deep and shallow multi-resolution feature fusion network consists of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature, so that the feature size of the deep layer and the shallow layer output is consistent, and a stacking operation is directly performed in the channel dimension; The defect classification module uses a trained defect classification network to extract features of defects for classifying the defects.

2. The defect classification system according to claim 1, characterized in that: The average pooling layer reduces the size of features by calculating the average value within the pooling window.

3. The defect classification system according to claim 1, characterized in that: Commonly used feature extraction networks are ResNet, DenseNet, and VGGNet networks.

4. The defect classification system according to claim 1, characterized in that: The deformable convolution layer consists of two parallel convolution layers that share feature inputs; One of them uses a conventional convolutional layer, where the convolution kernel size used by the conventional convolutional layer is 1 and the stride is 1; the other path uses an additional convolutional layer to learn the position offset in the x and y directions, and the number of output channels is twice that of the output of the conventional convolutional layer.

5. The defect classification system according to claim 1, characterized in that: The dimensions of the features of different resolutions are , where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature. The h, j, and c of deep features and shallow features are different; After the features of different resolutions are input into the regular convolution layer in the deformable convolution layer, the feature dimension of the first regular output feature is , where d represents the ratio of compressing the input channel; After the features of different resolutions are input into the additional convolution layer in the deformable convolution layer, the feature dimension of the first additional output feature is ; The first regular output feature and the first additional output feature are combined to obtain the second output feature, whose feature dimension is ; The second output feature is input into an average pooling layer to compress the second output feature size, and the feature dimension of the third output feature is obtained as , where k represents the size of the pooling window; different k are selected for deep features and shallow features, so that the deep features and shallow features are and The same means that the feature sizes of the deep features and the shallow features are consistent, and stacking operations are performed directly in the channel dimension.

6. The defect classification system according to claim 1, characterized in that: Perform a deformable convolution on the feature r, and the output is: Among them, R represents the feature space of the input features, is an enumeration of all positions in the feature space, is the current pixel position, Indicates the position offset, represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature, and f(·) corresponds to the pixel value at the specified position of the output feature; Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows: in, represents the sample position after the shift, represents the position after the shift, G(·,·) is the bilinear interpolation weight, where in( , ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, ( , ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

7. The defect classification system according to claim 1, characterized in that: The method for cutting defect samples is to count the length of the short side of the defect area, cut the defect area as the defect sample according to the result, and cut the non-defect area with the same size as the normal sample.

8. The defect classification system according to claim 7, characterized in that: The normal samples and defective samples are taken from the same picture to ensure that the defective samples are consistent with the normal samples in other factors such as lighting and shooting angle except for the presence or absence of defects.

9. The defect classification system according to claim 1, characterized in that: The method for establishing the feature data set is to manually label the defect samples and normal samples to establish four types of label data: warp defects, weft defects, hole defects or no defects.

10. The defect classification system according to claim 1, characterized in that: The calibrating the defect area position information includes calibrating the coordinates of the upper left corner and the lower right corner of the defect area.

11. A texture image defect classification method based on multi-resolution feature fusion, characterized in that: The defect classification method is to directly fuse the shallow features and deep features of the defects with different resolutions, and classify the defects after fusion. The defect classification method includes the following steps: Step (a-1): using an image acquisition module to collect images of defects to be tested; Step (a-2): using a data processing module to calibrate the defect area position information of the defect image to be tested, intercepting defect samples to be tested, inputting the defect samples into a common feature extraction network to obtain features of different resolutions, and establishing a feature data set, wherein the features include deep features and shallow features; Step (a-3): using the deep and shallow multi-resolution feature fusion network in the defect classification network training module to extract the features of each sample in the feature data set, and perform defect classification network training; the deep and shallow multi-resolution feature fusion network is composed of a deformable convolution layer and an average pooling layer, the deformable convolution layer is used to compress the number of channels of the input feature; the average pooling layer is used to reduce the size of the feature, so that the feature size of the deep layer and the shallow layer output is consistent, and a stacking operation is directly performed in the channel dimension; Step (a-4): using a defect classification module to classify defects.

12. The defect classification method according to claim 11, characterized in that: The average pooling layer reduces the size of features by calculating the average value within the pooling window.

13. The defect classification method according to claim 11, characterized in that: The commonly used feature extraction networks are ResNet, DenseNet, and VGGNet networks.

14. The defect classification method according to claim 11, characterized in that: The feature dimensions of the features of different resolutions are , where h is the length of the feature, j is the width of the feature, and c is the number of channels of the feature. The h, j, and c of deep features and shallow features are different.

15. The defect classification method according to claim 11, characterized in that: The implementation steps of the deep and shallow multi-resolution feature fusion network in step (a-3) are as follows: Step (a-31): Input the features of different resolutions into the regular convolution layer in the deformable convolution layer to obtain the first regular output feature, whose feature dimension is , where d represents the ratio of compressing the input channel; Step (a-32): Input the features of different resolutions into an additional convolution layer in the deformable convolution layer, and obtain the feature dimension of the first additional output feature as , where d represents the ratio of compressing the input channel; Step (a-33): Combine the first regular output feature and the first additional output feature to obtain a second output feature, whose feature dimension is ; Step (a-34): Input the second output feature into an average pooling layer, compress the second output feature size, and obtain the feature dimension of the third output feature as , where k represents the size of the pooling window; different k are selected for deep features and shallow features, so that the deep features and shallow features are and The same means that the feature size of the deep feature is consistent with that of the shallow feature output, and the stacking operation is performed directly in the channel dimension; There is no particular order for step (a-31) and step (a-32), and they can be performed one after the other or in parallel.

16. The defect classification method according to claim 11, characterized in that: The method for obtaining the defect sample to be tested in step (a-2) comprises the following steps: Step (b-1): calibrate the defect image to be tested collected in step (a-1), and convert the defect area position information into Four values ​​are calibrated, among which , Indicates the coordinates of the upper left corner of the defect area, Indicates the coordinates of the lower right corner of the defect area; Step (b-2): Take the defect sample to be tested: Calculate the short side of the defect area, count the length of the short side, and select the appropriate image size based on the result; Step (b-3): Use a box of a selected size to intercept the defect area and the normal area to obtain the defect sample to be tested and the normal sample respectively.

17. The defect classification method according to claim 11, characterized in that: The deformable convolution layer consists of two parallel convolution layers that share feature inputs; One of them uses a conventional convolutional layer with a kernel size of 1 and a stride of 1; The other path uses an additional convolutional layer to learn the position offset in the x and y directions, and the number of output channels is twice that of the regular convolutional layer output; Perform a deformable convolution on the feature r, and the output is: Among them, R represents the feature space of the input features, is an enumeration of all positions in the feature space, is the current pixel position, Indicates the position offset, represents the weight matrix, r(·) represents the pixel value at the specified position of the input feature, and f(·) represents the pixel value at the specified position of the output feature; Among them, since the offset is usually a decimal, the position p after the offset is realized by bilinear interpolation, and the formula is as follows: in, , represents the position after offset, G(·,·) is the bilinear interpolation weight; in( , ) represents the x-axis and y-axis coordinate values ​​corresponding to point a, ( , ) represents the x-axis and y-axis coordinate values ​​corresponding to point b.

Images