Color space conversion method and channel split convolution method for improving model performance

Abstract

The present application relates to a color space conversion method and a channel segmentation convolution method for improving model performance, which converts the commonly used RGB color space to XYZ color space for feature extraction, and improves the existing conversion matrix during conversion, so that the content of the original RGB three channels remains unchanged, and the content of another two channels is appropriately added in each channel, so that the information of each channel is more abundant, the pixel value of the noise point can be enlarged, the original noise can be disabled or enlarged to be better recognized by the model, and the feature extraction is more beneficial. Based on the convolutional neural network model, the model construction for detecting fabric defects is improved, the n-channel original data input by the convolution layer is re-segmented by the improved color space conversion method to obtain new n-channel data which is more beneficial to the feature extraction of the convolution layer, and the performance of the model is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a color space conversion method for improving model performance and a channel segmentation convolution method. BACKGROUND

[0002] In the prior art, a deep convolutional neural network model is widely used to detect defects of a fabric, which usually involves image feature extraction, image segmentation, target detection and the like. As known, a convolution operation has a deep influence and wide application in image feature extraction and channel dimension conversion, and is present in various deep learning model frameworks.

[0003] In optimizing a model, in addition to optimizing the model itself, various optimizations are also performed on input data, such as image enhancement and sample expansion. Data with obvious feature difference, sufficient samples and uniform distribution can often train a better model. The quality of the parameters of the model itself determines the lower limit of the model, and the quality of the samples to be tested directly determines the upper limit of the model. The role of the convolution layer in the model is to extract m channel features from given n channel data. We regard the convolution as a small model, whose input data is n channel data and output data is m channel data. The factors affecting the quality of the extracted features are related to the convolution kernel parameters, and therefore the convolution kernel parameters are optimized step by step through the back propagation of the loss function of the model. The quality of the convolution kernel parameters only determines the lower limit of the feature extraction capability of the convolution layer, and the real upper limit is still the original input sample data, i.e. the n channel data input into the convolution layer. However, through analysis of the application of the convolution layer in various models in the prior art, it can be found that the input channel data of each convolution layer is fixed in a certain sense. The input changes with the update of the convolution kernel parameters of the previous layer, but this change is local rather than globally optimal.

[0004] Before the convolution operation in the prior art, an RGB three-channel color image is usually input, and the original color image is divided into r, g and b three channels by default during the convolution operation. Three convolution kernels are used to filter and extract features from the three channels respectively, and a feature map is obtained by channel splicing. The main disadvantages of the image processing in the prior art are as follows:

[0005] (1) The default division into r, g and b three channels before the convolution operation is not the optimal choice, and the accuracy of the model detection still has room for performance improvement;

[0006] (2) The extraction of image features is mostly based on the RGB color space by default. The information of pictures in different color spaces is obviously different, and different color spaces can be converted into each other. Therefore, fully utilizing the conversion between different color spaces and the characteristics and advantages of different color spaces can also improve the performance of the model.

[0007] In view of the above-mentioned prior art, there is still room for further optimization, the inventors improve the default channel segmentation method before convolution operation through the conversion comparison of different color spaces, and comprehensively improve the model performance for detecting fabric defects, thus the present case is born. SUMMARY

[0008] The present application first discloses a color space conversion method for improving model performance, which converts the default RGB color space into XYZ color space, and improves the conversion matrix involved in the traditional XYZ color space, thereby enhancing the distinctiveness of the difference between the target feature and the background feature, and being more conducive to feature extraction. After XYZ color space conversion, the noise point value added at the origin is magnified, so that the noise point is visualized, can be well learned and recognized by the model, removes the influence of noise, and improves the detection accuracy of the model.

[0009] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0010] The color space conversion method for improving model performance includes the following contents:

[0011] The RGB color space is converted into XYZ color space, that is, the input picture is channel separated and divided into r, g and b three channels, then the divided r, g and b three channels are multiplied by an n*n XYZ space conversion matrix M to combine into three new channels, the elements in the same row and column in the conversion matrix M are all 1, and other elements are all less than 1.

[0012] Further, the elements in the conversion matrix M other than the elements in the same row and column are greater than 0 and less than 1.

[0013] Further, the elements in the conversion matrix M other than the elements in the same row and column are greater than 0 and less than 0.5.

[0014] Further, the conversion matrix The three new channels are: r+0.2g+0.1b, 0.2r+g+0.1b, 0.1r+0.2g+b.

[0015] Based on the above-mentioned color space conversion method, the present application further discloses a channel segmentation convolution method for improving model performance, which includes the following contents:

[0016] Step 1: constructing a convolutional neural network model and constructing an n*n unit conversion matrix M, n is the input dimension of the corresponding convolutional layer in the convolutional neural network model, the elements in the same row and column in the matrix M are 1, and the other elements are 0, M matrix is included in the model learning parameters Parm(M), wherein Parm represents the model parameter set;

[0017] Step 2: multiply the original input RGB image with the matrix M to perform initial conversion of the channel;

[0018] Step 3: continuously update Parm(M) through the convolutional neural network model back propagation until the optimal new matrix M' is obtained;

[0019] Step 4: before the convolution kernel is used to convolve the picture in the convolutional layer of the trained convolutional neural network model, the input RGB image is multiplied with the new matrix M' to obtain a new channel after redivision, and then the convolution kernel of the layer is used to convolve the picture of the new channel.

[0020] Further, the elements in the same row and column in the new matrix M' are all 1, and the other elements are all greater than 0 and less than 1.

[0021] Further, in the new matrix M', except the elements in the same row and column, the other elements are all greater than 0 and less than 0.5.

[0022] The present application fully evaluates the advantages and disadvantages of different color spaces, and finally obtains that the conversion of the commonly used RGB color space in traditional picture processing to XYZ color space for feature extraction is helpful to improve the accuracy of model detection. When performing color space conversion, the present application also improves the conversion matrix commonly used in the existing XYZ color space, so that the content of the original RGB three channels remains unchanged, and the content of the other two channels is appropriately added in each channel, so that the information of each channel is more abundant. To a certain extent, the noise point pixel value can be amplified, so that the original noise is ineffective or becomes large and can be better recognized by the model. After increasing the content of other channels, the difference between the converted background and fabric defect features can be increased, which is more conducive to feature extraction. Using the conversion idea of the color space, the improved color space conversion method is used in the model construction based on the convolutional neural network model to detect fabric defects. The n-channel original data input into the convolutional layer is re-divided by using the improved color space conversion method to obtain new n-channel data that is more conducive to feature extraction of the convolutional layer, which helps to improve the performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The present application color space and other color space fabric defect detection effect comparison chart;

[0024] Figure 2 A convolutional channel segmentation flowchart in the prior art;

[0025] Figure 3 A channel segmentation convolutional flowchart of the present application;

[0026] Figure 4 A structural diagram of a prior art model convolution and a channel segmentation convolution of the present application;

[0027] Figure 5 A Lenet5 model architecture diagram. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0029] The RGB color space is a commonly used color space in the prior art, but using RGB cannot accurately define colors, because different devices display the same RGB, which is also different in the human eye. The RGB color space is divided into three single channels of r, g and b, which are independent of each other and do not affect each other. The RGB color space can more truly reflect the information of a color image. In the field of fabric defect detection, a detection model is used to automatically identify fabric defects. In order to improve the accuracy of the detection model, the advantages of different color spaces are fully utilized to realize the conversion between color spaces, which helps to improve the precision of the model. Therefore, the inventors have conducted comparative research on feature extraction of pictures in different color spaces, and have converted the RGB color space to the XYZ color space and to other color spaces (such as the HSV color space) for comparison. It can be seen through comparison that the conversion from RGB to XYZ color space is a certain linear matrix transformation, while the conversion from RGB to other color spaces (such as the HSV color space) involves nonlinear conversion and is directly related to the size of the pixel value of a certain channel. Therefore, the conversion process is easily affected by the specific pixel value of a certain point, has high uncertainty and strong sensitivity, and is not easy to analyze. Therefore, only when the RGB color space is converted to the XYZ color space for feature extraction, the accuracy of the detection model can be maximized.

[0030] Therefore, the inventors propose a color space conversion method for improving the performance of a model, which specifically includes the following contents: performing channel separation on an input RGB picture to separate it into three channels of r, g and b, then multiplying the separated three channels of r, g and b by an n x n XYZ space conversion matrix M to combine them into three new channels. The elements in the same row and column in the conversion matrix M must all be 1, and other elements are all less than 1.

[0031] The conversion matrix multiplied in the prior art XYZ color space conversion is not the same as the conversion matrix M described above in the present application. In the conversion matrix in the prior art, each element is less than 1, and some elements are negative. Using the prior conversion matrix cannot analyze whether the feature difference after color space conversion is obvious, cannot directly determine whether the noise value is increased or reduced, and cannot well improve the detection precision of the fabric flaw detection model. The conversion matrix given in the prior art about standards BT2020 and BT709 is as follows:

[0032]

[0033] And the conversion matrix given in the prior art about standard DCI-P3 is as follows:

[0034]

[0035] As can be seen, if the above prior conversion matrix is used to convert the RGB color space, the content of r, g and b in each channel will be changed, which will affect the display of image information color. In the improved conversion matrix M of the present application, the elements on the diagonal are all 1, which can ensure that the r content of the x channel is the highest and unchanged after conversion, the g content of the y channel is the highest and unchanged, and the b content of the z channel is the highest and unchanged, which can preserve the characteristic advantages of the RGB three channels to a certain extent, and also appropriately increases the content of the other two channels in each channel, so that each channel contains r, g and b three channel components at the same time, which is closer to the original input color image information, and enriches the three channel feature information.

[0036] Taking a 3x3 conversion matrix as an example, the conversion of RGB and XYZ color spaces is carried out according to the following formula:

[0037]

[0038] After color space conversion, three new channels are obtained, which are r+0.2g+0.1b, 0.2r+g+0.1b and 0.1r+0.2g+b.

[0039] The effects before and after conversion are compared as follows:

[0040] (1) Comparison from the aspect of target feature and background feature difference visualization;

[0041] Suppose there are two pixel points, one is a background point and one is a flaw point, the three channel values of the background point are S 背 =(a,b,c); and the three channel values of the flaw point are S 瑕 =(d,e,f).

[0042] When the color space is not converted, the difference between the two points in the original image is Δ1=(a-d,b-e,c-f).

[0043] After conversion into XYZ color space by the 3x3 conversion matrix given above, the three-channel values of the two pixel points are respectively:

[0044]

[0045] The difference between the two points after color space conversion can be obtained: Δ2=[1.0(a-d)+0.2(b-e)+0.1(c-f), 0.2(a-d)+1.0(b-e)+0.1(c-f), 0.1(a-d)+0.2(b-e)+1.0(c-f)].

[0046] By comparing Δ1 and Δ2, it is not difficult to find that after conversion by the conversion matrix, the feature difference between the defect and the background is more obvious, which is more conducive to feature extraction.

[0047] (2) Comparison from the aspect of anti-noise interference layer;

[0048] Suppose the original pixel point value of a certain point is: S 原 =(a, b, c); after adding noise to the point, the noise point value is: Snoise=(d, e, f).

[0049] When the color space is not converted, the difference between the two points in the original image is: Δ1=(a-d, b-e, c-f).

[0050] After the point is affected by noise, the pixel point value becomes: S0=(a+d, b+e, c+f).

[0051] After XYZ color space conversion, the pixel point value affected by noise is:

[0052]

[0053] As can be seen, after XYZ color space conversion, the noise point value added to the original point is amplified, making the noise point visible, which can be well learned and recognized by the model, and the influence is removed.

[0054] In order to verify that the color space conversion method of the application can improve the target detection index MAP of the detection model, the inventors constructed a Yolov7 model for detecting fabric defects, and used the model to complete the training and testing of the model without color space conversion, by the existing XYZ color space conversion matrix, and by the color space conversion method of the application, and compared the detection effects of fabric defects under the above different methods (as shown in Figure 1 ).

[0055] Figure 1The upper left corner (a) is the real defect detection graph under the original label of four pieces of fabric, the upper right corner (b) is the detection effect graph of the same four pieces of fabric in the original RGB color space using the Yolov7 model, the lower left corner (c) is the detection effect graph of the same four pieces of fabric in the XYZ color space converted from the RGB color space using the Yolov7 model according to the prior art, and the lower right corner (d) is the detection effect graph of the same four pieces of fabric using the Yolov7 model according to the color space conversion method of the application. Among them, Figure 1 (d) in the following steps:

[0056] S1) Convert the original r, g, b three channels into r+0.2g+0.1b, 0.2r+g+0.1b, 0.1r+0.2g+b three new channels according to the color space conversion method of the application.

[0057] S2) Load the converted data set, and then perform transform transformation to crop each photo into the shape of (640, 640), and then convert it into a tensor, and then perform DataLoader with batch_size=16.

[0058] S3) Parameter setting:

[0059] epochs=100, batch_size and data DataLoader=16, learning_rate=0.0001. The optimizer is selected as Adam.

[0060] S4) Train and test the Yolov7 model using the converted data set, and use the tested model to detect defects in the four pieces of fabric, and output Figure 1 the detection effect graph of (d) in the.

[0061] Through actual comparison and verification, the results show that without color space conversion, directly using the Yolov7 model for detection ( Figure 1 (b) in the), MAP=34.2%; converting the RGB color space to the XYZ color space according to the prior art to complete the detection ( Figure 1 (c) in the), MAP=37.9%; converting the color space according to the application to complete the detection ( Figure 1 (d) in the), MAP=38.3%, which shows that through the conversion of the XYZ color space, the MAP index of the original RGB color space is indeed improved, and through the effect graphs shown in (b) and (c) in the Figure 1 it can be seen that through the conversion of the XYZ color space, defects that cannot be detected in the original model are found. Again from Figure 1Comparing the results of the two different color space conversion methods (c) and (d), the method of the present invention improves the MAP index more and detects more defect features, indicating that the method of the present invention is effective.

[0062] This invention also applies the aforementioned color space conversion method to channel segmentation in convolution operation feature extraction, thereby improving model performance. An adaptive channel segmentation convolution module is proposed, which is plug-and-play and can be inserted before any convolutional layer in a convolutional neural network model to re-segment the channels before performing convolution processing. Figure 4 As shown, the left figure is the conventional model detection process, the middle figure is the flowchart of inserting the adaptive channel segmentation convolution module of the present invention before the first convolutional layer, and the right figure is the flowchart of inserting the adaptive channel segmentation convolution module of the present invention before any convolutional layer. Figure 4 For simplification, the left, middle, and right diagrams only show the structure of one layer of the model; the remaining layers are represented by “…” symbols and omitted. Figure 4 In this context, n represents the number of feature map channels input to the module.

[0063] Conventional convolutional channel segmentation in existing technologies, such as Figure 2 As shown, this invention utilizes an adaptive channel segmentation convolution module to segment the convolution channels, as follows: Figure 3 As shown, the adaptive channel splitting convolution module uses a newly created n×n identity transformation matrix M to convert the original three RGB channels into three new channels.

[0064] This invention relates to a channel segmentation convolution method for improving model performance, including the following:

[0065] Step 1: Construct a convolutional neural network model (in this embodiment, the model is constructed as follows) Figure 5 The LeNet-5 model shown is used. A new n×n identity transformation matrix M is created (n is the input dimension of the corresponding convolutional layer in the convolutional neural network model, i.e., the number of channels; in this embodiment, n=3 is used as an example). In this embodiment, the elements in the newly created identity transformation matrix M that have the same row number and column number are all 1, and the other elements are all 0. The M matrix is ​​included in the model learning parameters, i.e., Parm(M), where Parm represents the model parameter set.

[0066]

[0067] Step 2: Multiply the original input RGB image with the transformation matrix M to perform initial channel transformation;

[0068]

[0069] Step 3: Continuously backpropagate through the convolutional neural network model to update Parm(M), making its channel conversion more conducive to improving model performance, until the optimal new matrix M′ is obtained; after self-learning, the elements in the new matrix M′ with the same row number and column number are all 1, and other elements are all greater than 0 and less than 1. According to actual simulation tests, the best performance is achieved when all elements in the new matrix M′ except for those with the same row number and column number are greater than 0 and less than 0.5.

[0070]

[0071] Step 4: Before convolving the image with convolution kernels in the convolutional layer of the trained convolutional neural network model, multiply the input RGB image with the new matrix M′ to obtain the newly divided channels new1, new2, and new3. Then, use the convolution kernels of this layer to convolve the image of the new channels.

[0072]

[0073] To verify the effectiveness of the proposed channel segmentation convolution method for image feature extraction on different models and datasets, the inventors constructed a LeNet-5 model different from the model used in the color space conversion verification above. They also selected 1,000 images each of cats, dogs, and pandas as a dataset and compared the convolution methods of existing models with the CIFAR10 dataset.

[0074] Among them, with Figure 5 Taking the LeNet-5 model shown as an example, a dataset of 3000 images was collected, consisting of 1000 images each of cats, dogs, and pandas. The specific verification method is as follows:

[0075] S1) Use the MakeSense tag creation tool to create tags.

[0076] S2) Construct the LeNet-5 model.

[0077] Specifically, it includes the following steps:

[0078] (1) First, use 6 5*5 convolution kernels to convolve the 3-channel 32*32 image, and the result is 6 28*28 feature maps;

[0079] (2) Perform 2*2 max pooling on the 28*28 feature map to obtain a 14*14 feature map;

[0080] (3) Activated via the ReLU layer;

[0081] (4) using 16 5*5 convolution kernels to convolve the 6-channel 14*14 picture, resulting in 16 10*10 feature maps;

[0082] (5) 2*2 maximum pooling is performed on the 10*10 feature map to obtain a 5*5 feature map;

[0083] (6) activated through the ReLu layer;

[0084] (7) using 120 5*5 convolution kernels to convolve the 16-channel 5*5 picture, resulting in 120 1*1 feature maps;

[0085] (8) activated through the ReLu layer;

[0086] (9) the down-sampled picture of (None, 1, 1, 120) is stretched into the shape of (None, 120);

[0087] (10) 120 features are mapped through the fully connected layer to 84;

[0088] (11) finally, the 84 features are mapped to 3 (corresponding to the number of target categories) through the softmax layer.

[0089] S3) insert the adaptive channel split convolution module of the application at the first layer of the convolution layer, and build a Lenet5-new_split model.

[0090] Specifically, through the adaptive channel split convolution module, the input image channel is first re-divided, and then 6 5*5 convolution kernels are used to convolve the newly divided 3-channel 32*32 picture, resulting in 6 28*28 feature maps; the subsequent steps are the same as steps (2) to (11) in S2) above.

[0091] S4) import the collected cat, dog, and panda three-classification data set, and preprocess the data set: first, transform the data set, crop each photo to (32, 32) shape, then convert to tensor, and then use batch_size=16 to DataLoader.

[0092] S5) parameter setting:

[0093] epochs=100, batch_size and data DataLoader=16, learning_rate=0.0001, and optimizer is Adam;

[0094] The loss calculation method uses cross-entropy loss (CrossEntropyLoss), which has:

[0095]

[0096] wherein Loss represents a sample loss, a j represents a softmax function output, y j represents a real label of the jth sample, and is in one-hot form.

[0097] S6) Model training: initializing all gradients to 0, and obtaining a prediction label outputs through model training; calculating a loss through the outputs and the real label labels; then performing back propagation to update the gradients; and calculating an accuracy after each round of training:

[0098] accuracy = 100.0 * correct / total

[0099] wherein correct represents a number of samples in which the prediction label outputs is equal to the real label labels, and total represents a total number of samples.

[0100] S7) Comparison of experimental effects:

[0101] Directly using the original LeNet-5 model to test the cat, dog and panda three-classification data set, an accuracy of 71% is obtained; using the Lenet5-new_split model with the adaptive split module of the application to test the cat, dog and panda three-classification data set, an accuracy of 79% is obtained.

[0102] Comparative verification is performed again by replacing the data set, and this time, the CIFAR10 data set is used, the cat, dog and panda three-classification data set is replaced with the CIFAR10 data set and reimported according to the content given in S4) above, and the subsequent steps are performed in the same way as S4) to S6) above. The experimental effects after replacing the data set are compared: directly using the original LeNet-5 model to test the CIFAR10 data set, an accuracy of 60% is obtained; using the Lenet5-new_split model with the adaptive split module of the application to test the CIFAR10 data set, an accuracy of 63% is obtained.

[0103] As can be clearly seen from the above, by using the channel segmentation convolution method given in the application, each layer of convolution layer can automatically find the channel segmentation method most suitable for its operation for different data sets and different models, and the performance of the model is better improved.

[0104] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A channel split convolution method for boosting model performance, characterized in that, The application comprises the following contents: Step 1: constructing a convolutional neural network model and an n*n unit conversion matrix M, n is the input dimension of the corresponding convolutional layer in the convolutional neural network model, the elements of the same row and column in the matrix M are all 1, and the other elements are all 0, the M matrix is included in the model learning parameters Parm(M), wherein Parm represents the model parameter set; Step 2: multiplying the original input RGB image with the matrix M to perform initial conversion of the channel; Step 3: continuously updating Parm(M) through the convolutional neural network model back propagation until the optimal new matrix M' is obtained; Step 4: before the trained convolutional neural network model is used to perform convolution on the picture by using the convolution kernel, the input RGB image is multiplied with the new matrix M' to obtain a new channel after redivision, and then the new channel picture is convolved by using the convolution kernel of the layer.

2. The method of claim 1, wherein: The elements of the same row and column in the new matrix M' are all 1, and the other elements are all greater than 0 and less than 1.

3. The method of claim 2, wherein: In the new matrix M', except the elements of the same row and column, the other elements are all greater than 0 and less than 0.5.

Images