An efficient information- preserving down-sampling method and down-sampling module

Abstract

The application provides an efficient information fidelity down-sampling method and a down-sampling module, and is a new effective down-sampling method in a segmentation task; the module uses Haar wavelet transform to reduce the spatial size, and simultaneously increases the number of feature channels, so that the information of the space is not lost; a 1*1 pixel-by-pixel convolution operation is used to learn representative features, so that the features of the image are extracted with a small number of learning parameters, and the segmentation quality of semantic segmentation is improved; the performance of the segmentation model is enhanced, and the function of evaluating the features after down-sampling is realized. The general down-sampling module provided by the application directly replaces the pooling layer or the stride convolution layer without increasing the calculation amount, and is integrated into the current architecture of semantic segmentation; the application provides a new metric index feature entropy, which is used for measuring the quality of the features after sampling, and evaluating the size of the uncertainty of the output feature map.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to an efficient information fidelity down-sampling method and a down-sampling module. BACKGROUND

[0002] In recent years, with the rapid development of deep convolutional neural networks, semantic segmentation has also made great progress. As the most basic component of deep convolutional neural networks, down-sampling is often used to change the resolution of the input. Therefore, down-sampling can well expand the receptive field of the neural network, while effectively reducing the computational overhead. The down-sampling methods commonly used in deep convolutional neural networks, such as max pooling, stride convolution and average pooling, will cause the loss of important information such as texture and boundary in semantic segmentation. In order to effectively alleviate the loss of information, and establish the relationship between the classification label and the down-sampled feature, a method of transmitting more information to the multi-scale feature fusion subnet or the encoder subnet is proposed. Some works based on this method can partially alleviate the loss of information due to down-sampling, and aggregate more information, so that representative feature maps are extracted. However, for some simple feature fusion methods such as feature map concatenation or direct addition, too much invalid information will be input into the network, which hinders the learning of effective features.

[0003] For segmented objects with occlusion between foreground and background or small scale, the lost information such as texture and boundary is very important. However, for the current work, it still faces great challenges to recover the lost information at the subsequent decoding end.

[0004] Haar wavelet transform is mainly used in image compression, reconstruction, decomposition and denoising in early image processing research. Haar wavelet transform has the advantage of fast multi-scale signal decomposition.

[0005] Deep convolutional networks learn features layer by layer and use the learned features for final inference. From the perspective of information quantity, it can be understood as a process of continuously reducing information entropy. Information entropy is often used to evaluate the uncertainty of images or signals in communication systems in previous studies. The main role of neural networks in segmentation tasks is to reduce the uncertainty between input labels and images, and this uncertainty needs to be evaluated. SUMMARY

[0006] The technical problem to be solved by the application is to provide an efficient information fidelity down-sampling method and a down-sampling module for enhancing the performance of a segmentation model and evaluating the features after down-sampling.

[0007] The technical scheme adopted by the present application to solve the above technical problems is: an efficient information fidelity down-sampling method, comprising the following steps:

[0008] S1: inputting an image or a feature map into a lossless feature encoding part of a high-fidelity down-sampling module for feature encoding, and decomposing by Haar wavelet transform;

[0009] S2: inputting the encoded features in step S1 into a feature learning part of the down-sampling module to learn representative features, change the channel number of the feature map and filter redundant information;

[0010] S3: using a feature quality measurement index, feature entropy, to evaluate the down-sampled feature map obtained in step S2;

[0011] S4: defining the loss function in the training process as the sum of the generalized Dice loss and the cross-entropy loss.

[0012] According to the above scheme, in step S1, the specific steps are:

[0013] S11: assuming that the size of the input feature map is CxHxW, C, H and W represent the channel number, the height and the width of the feature map respectively; performing row filtering along the column direction, and then performing down-sampling on each row;

[0014] S12: performing column filtering on the result obtained in the previous step along the row direction, and then performing down-sampling on each column; obtaining three detail components in the vertical, horizontal and diagonal directions respectively and an approximate component; and outputting a feature map with a size of 4CxH / 2xW / 2.

[0015] According to the above scheme, in step S1, the specific steps are:

[0016] The definition formula of the Harr wavelet transform is as follows:

[0017]

[0018] φ j,k (x) is defined as follows:

[0019]

[0020] φ 0,0 (x) is defined as follows:

[0021]

[0022] According to the above scheme, in step S2, the specific steps are:

[0023] S21: the input feature map is first subjected to feature extraction by the convolution layer of the feature learning part;

[0024] S22: The batch normalization layer of the feature learning part performs normalization operation on each pixel point in the feature map;

[0025] S23: After the ReLU activation of the feature learning part, four final down-sampling feature maps are obtained.

[0026] According to the above scheme, in the step S3, the feature entropy index is used for quantitatively evaluating the quality of the sampled features, measuring the number of features learned by the convolution layer of the feature learning part, and evaluating the size of the uncertainty of the output feature map; the smaller the obtained feature entropy index is, the better the model is in segmenting the feature map.

[0027] Supposing that the feature map obtained by down-sampling is F, there are C feature channels, and the resolution after the bilinear interpolation is HxW; P represents the final prediction result of the segmentation model, and the size is HxW; the total number of pixels of P is represented by M, and the size is HxW; and the feature entropy is:

[0028]

[0029] According to the above scheme, in the step S4, supposing that g, p, N and C represent the gold standard of expert annotation, the prediction result, the number of image elements and the actual label of each pixel respectively, the loss function in the training process is:

[0030]

[0031] In the formula, the first term and the second term are cross-entropy loss and generalized Dice loss function.

[0032] An efficient information fidelity down-sampling module, including a down-sampling module including a lossless feature encoding part and a feature learning part; the lossless feature encoding part is used for inputting an image or a feature into a low-pass filter and a high-pass filter to extract low-frequency information and high-frequency information; the feature learning part includes a standard convolution layer, a batch normalization layer and a ReLU activation layer; the feature learning part is used for learning representative features, changing the number of channels of the feature map and filtering redundant information.

[0033] A computer storage medium, which stores a computer program executable by a computer processor, the computer program executing an efficient information fidelity down-sampling method.

[0034] The beneficial effects of the present application are:

[0035] 1.A high-efficiency information-fidelity down-sampling method and a down-sampling module are provided in the present application, which are designed for the purpose of preserving as much information as possible inside a deep convolutional neural network in semantic segmentation, and the Haar wavelet down-sampling module and the information-fidelity down-sampling method based on Haar wavelet transform are new effective down-sampling methods in segmentation tasks; the module uses Haar wavelet transform to reduce the spatial size while increasing the number of feature channels, so that the information of the space will not be lost; 1x1 pixel-wise convolution operation is used to learn representative features, and a small number of learning parameters are used to extract the features of the image, thereby improving the segmentation quality of semantic segmentation; and the performance of the segmentation model is enhanced and the function of evaluating the features after down-sampling is realized.

[0036] 2.The general down-sampling module provided in the present application includes a lossless feature encoding part and a feature learning part; the pooling layer or the stride convolution layer is directly replaced without increasing the amount of calculation, and is integrated into the current architecture of semantic segmentation; the lossless feature encoding part enables information to be preserved as much as possible at the backbone of the neural network, greatly improves the performance of various image segmentation models based on the convolutional neural network architecture, and significantly improves the segmentation quality of small-scale targets; the general down-sampling module alleviates the loss of important spatial information caused by down-sampling in the semantic segmentation task, effectively reduces the uncertainty of the information, easily learns representative features, and improves the overall performance of the image segmentation model based on the convolutional neural network architecture in terms of semantic objects (especially the segmentation of small-scale targets).

[0037] 3.The present application proposes a new metric index feature entropy for the purpose of measuring and confirming the number of features learned by the subsequent convolutional layer of the down-sampling module designed in the present application when applied in the deep convolutional neural network, which is used to measure the quality of the sampled features and evaluate the size of the uncertainty of the output feature map. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a structural schematic diagram of the down-sampling module of an embodiment of the present application.

[0039] Figure 2 FIG. 2 is a Haar wavelet transform structural schematic diagram of the lossless feature encoding part of an embodiment of the present application.

[0040] Figure 3 FIG. 3 is a structural and result schematic diagram of the general down-sampling method of an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0042] Referring to Figure 1 , the embodiments of the present application include two aspects: a high-efficiency information-fidelity down-sampling method and a feature metric index.Figure 1 The two solid boxes are two parts of the method: lossless feature coding and feature learning.

[0043] The efficient information fidelity down-sampling method comprises the following steps:

[0044] Step 1: input an image or a feature map into a lossless feature coding part of the high-fidelity down-sampling module for feature coding; the lossless feature coding part is to input the input image or feature into a low-pass filter and a high-pass filter to extract low-frequency information and high-frequency information of the image.

[0045] Suppose that the size of the original input feature map is CxHxW, C, H and W represent the channel number, the height and the width of the feature map, respectively. Figure 2 The Haar wavelet transform structure diagram of the lossless feature coding part of the application is shown in the figure, and the decomposition process of the Haar wavelet on the image can be seen from the filtering process shown in the figure.

[0046] First, row filtering is performed along the column direction, and then each row is down-sampled; then, column filtering is performed on the result obtained in the previous step along the row direction, and then each column is down-sampled; finally, three detail components in the vertical, horizontal and diagonal directions and an approximate component are obtained. The definition formula of the Harr wavelet transform is as follows:

[0047]

[0048] where φ j,k (x) is defined as follows:

[0049]

[0050] φ 0,0 (x) is defined as follows:

[0051]

[0052] After the Haar wavelet transform, 4CxH / 2xW / 2 output is obtained. This process reduces the resolution of the feature map by half without losing information, and at the same time increases the channel number of the feature map by four times.

[0053] Step 2, the encoded features in step 1 are input into the feature learning part to filter and change the number of channels of redundant information;

[0054] After passing through the lossless feature encoding part, the obtained feature map enters the feature learning part to learn representative features, change the number of channels of the feature map and filter redundant information. The feature learning part includes three parts: a standard convolution layer, a batch normalization layer and a ReLU activation layer. The convolution layer is used for convolution operation to learn representative features and change the number of channels; the convolution operation is used to filter redundant information, and the number of channels is changed for calculation with other feature maps. The input feature map is first subjected to feature extraction by the convolution layer, then the batch normalization layer is subjected to normalization operation on each pixel point in the feature map, and finally the ReLU activation is subjected to obtain the final four down-sampling feature maps.

[0055] Step 3, the feature map obtained in step 2 is evaluated by using the proposed metric indicator-feature entropy;

[0056] The feature entropy indicator is used to quantitatively evaluate the quality of the sampled features, evaluate the size of the uncertainty of the output feature map, and measure the number of features learned by the convolution layer of the feature learning part. The smaller the obtained feature entropy indicator is, the better the model can segment the feature map. The definition is as follows:

[0057]

[0058] In the formula, the down-sampled feature map is F, which has C feature channels and a resolution of HxW after bilinear interpolation. P represents the final prediction result of the segmentation model, which has a size of HxW. The total number of pixels of P is represented by M, which has a size of HxW.

[0059] Step 4, define the loss function in the training process;

[0060] The training loss function is the sum of the generalized Dice loss and the cross-entropy loss, and the calculation formula is defined as follows:

[0061]

[0062] In the formula, g, p, N and C represent the expert-annotated gold standard, the prediction result, the number of image elements and the actual label of each pixel, respectively. The first and second terms are the cross-entropy loss and the generalized Dice loss function.

[0063] The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made in accordance with the principles and design ideas disclosed by the present application are within the scope of protection of the present application.

Claims

1. An efficient information- preserving down-sampling method characterized by: The method comprises the following steps: S1: inputting an image or a feature map into a lossless feature encoding part of a high-fidelity down-sampling module for feature encoding, and decomposing by Haar wavelet transform; S2: inputting the encoded feature of step S1 into a feature learning part of the down-sampling module, learning representative features, changing the number of channels of the feature map, and filtering redundant information; S3: evaluating the down-sampled feature map obtained in step S2 by using a feature quality measurement index, i.e., feature entropy; the feature entropy index is used for quantitatively evaluating the quality of the sampled feature, measuring the number of features learned by the convolutional layer of the feature learning part, and evaluating the size of the uncertainty of the output feature map; the smaller the obtained feature entropy index is, the better the segmentation of the feature map by the model is; Suppose that the feature map obtained by down-sampling is F, there are C feature channels, and the resolution after bilinear interpolation is HxW; P represents the final prediction result of the segmentation model, and the size is HxW; the total number of pixels of P is represented by M, and the size is HxW; and the feature entropy is: ； S4: defining the loss function in the training process as the sum of the generalized Dice loss and the cross-entropy loss.

2. The method of claim 1, wherein: In the step S1, the specific steps are: S11: assuming that the size of the input feature map is , C , H and W represent the number of channels, the height and the width of the feature map, respectively; the row filtering is performed along the column direction, and then down-sampling is performed on each row; S12: column filtering on the result of the previous step along the row direction, then down-sampling column by column; three detail components in vertical, horizontal and diagonal directions and one approximation component are obtained; the output scale size is 4 .

3. The method of claim 1, wherein: In the step S1, the specific steps are: The definition formula of the Harr wavelet transform is as follows: ； The definition formula is as follows: ； The definition formula is as follows: 。 4. The method of claim 1, wherein: In the step S2, the specific steps are: S21: the input feature map is first subjected to feature extraction by the convolutional layer of the feature learning part; S22: the batch normalization layer of the feature learning part performs normalization operation on each pixel point in the feature map; S23: after the ReLU activation of the feature learning part, four final down-sampled feature maps are obtained.

5. The method of claim 1, wherein: The step S4 is provided with g , p , N and C respectively represent the expert-annotated gold standard, the prediction result, the number of image elements, and the actual label of each pixel, and the loss function in the training process is: ， The first term and the second term are the cross-entropy loss and the generalized Dice loss function.

6. A downsampling module for the efficient information- preserving downsampling method of any of claims 1 to 5, characterized by: The method comprises The down-sampling module comprises a lossless feature encoding part and a feature learning part; The lossless feature encoding part is used for inputting an image or a feature into a low-pass filter and a high-pass filter to extract low-frequency information and high-frequency information; The feature learning part comprises a standard convolutional layer, a batch normalization layer and a ReLU activation layer; the feature learning part is used for learning representative features, changing the number of channels of the feature map, and filtering redundant information.

7. A computer storage medium, characterized in that: The computer program stored therein can be executed by a computer processor, and the computer program executes the efficient information fidelity down-sampling method according to any one of claims 1 to 5.

Images