Feature map up-sampling method and system based on tensor dimension transformation and convolution

Abstract

The invention relates to a feature map up-sampling method and system based on tensor dimension transformation and convolution, and belongs to the field of computer image processing and artificial intelligence. The method comprises the following steps: inputting a feature map; performing convolution operation on the tensor of the feature map; carrying out dimension transformation and convolution on the feature map tensor obtained by the convolution operation; and outputting an up-sampling result obtained by dimension transformation and convolution. According to the method, up-sampling is achieved through convolution operation and dimension operation, the method is mainly suitable for the up-sampling process of small and medium-scale feature maps, compared with a traditional method, the calculation process is simpler, the calculation amount is lower than that of transposition convolution, the calculation precision is higher, and self-adaptive up-sampling of data can be achieved through learning. The method can be built in an encoder of a convolutional neural network image segmentation model, is used for carrying out up-sampling operation on a feature map, and can also be trained through a deep learning method to realize amplification and generation of specific image data.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing and artificial intelligence, and in particular relates to a feature map upsampling method and system based on tensor dimension transformation and convolution. Background Art

[0002] Convolutional Neural Networks (CNNs) play a vital role in image semantic segmentation. Image semantic segmentation is an important task in computer vision that aims to classify each pixel in an image so that it corresponds to the object category in the image. Semantic segmentation is of great significance in computer vision. It enables computers to understand the objects in an image and the relationships between them, thus supporting many applications. This task has important applications in autonomous driving, medical imaging, augmented reality, industrial quantitative detection and other fields.

[0003] The convolutional neural networks used in semantic segmentation tasks are mostly encoder-decoder configurations. The encoder is used to extract features from the input image and is composed of several convolution-downsampling feature extraction layers. The resolution of its output feature map gradually decreases and the number of channels gradually increases to achieve feature extraction. The decoder is composed of several convolution-upsampling layers. The resolution of its output feature map gradually increases and the number of channels gradually decreases. It outputs pixel classification results that are consistent with the resolution and number of channels of the input image to achieve semantic segmentation of the image.

[0004] In the model of the above structure, the upsampling operation in the decoder is crucial to the accuracy and performance of the model. In the image semantic segmentation network, the main role and significance of the upsampling layer is to increase the size of the image, so that the network's receptive field is larger, so that more contextual information can be captured. By mapping the low-resolution features in the feature map to the high-resolution space, a higher-quality feature map is obtained, which enables the network to better understand and segment the details and texture information in the image. In addition, the upsampling layer can also reduce the redundant information in the feature map by fusing the feature information of areas that may contain the same or similar semantic information, thereby improving the efficiency and accuracy of the network.

[0005] At present, the commonly used upsampling methods mainly include bilinear interpolation and transposed convolution. Among them, the bilinear interpolation algorithm is simple and has a fast calculation speed. It can effectively overcome the jagged edges and mosaic phenomenon of the image, but it is easy to cause the loss of image detail information, especially for the high-frequency signal. In addition, since bilinear interpolation only considers the influence of the four direct neighboring points around the sampling point, it ignores the influence of other adjacent pixels, which may cause the loss of some detail information.

[0006] Transposed convolution, also known as deconvolution or fractional-stride convolution, is a convolution operation that generates a higher-resolution feature map by filling the edges of a low-resolution feature map and then using a transposed convolution kernel to perform upsampling. Unlike bilinear interpolation, transposed convolution can achieve adaptive upsampling of data through learning, avoiding the problem of manually setting the upsampling rate. However, transposed convolution also has some problems. On the one hand, the calculation process is relatively complex and the amount of calculation is large, which makes the deployment of the model more difficult. On the other hand, the output result has a checkerboard effect, which reduces the accuracy of the model. Summary of the invention

[0007] In response to the above problems, the present invention provides a feature map upsampling method and system based on tensor dimensional transformation and convolution, which implements an upsampling operation for spatially enlarging the input feature map by transforming the image spatial dimension to the channel dimension and then performing convolution.

[0008] The first object of the present invention is to provide a feature map upsampling method based on tensor dimension transformation and convolution, the method comprising:

[0009] Input feature map;

[0010] Perform convolution operation on the tensor of feature map;

[0011] Perform dimension transformation and convolution on the feature map tensor obtained by the convolution operation;

[0012] Output the upsampled result of dimension transformation and convolution.

[0013] In the embodiment of the present invention, the convolution operation is a 1×1 convolution.

[0014] In the embodiment of the present invention, the step of performing dimension transformation and convolution on the feature map tensor obtained by the convolution operation includes:

[0015] Transform the dimension of the feature map tensor obtained by the convolution operation;

[0016] Perform group convolution operation on the feature map tensor obtained by dimension transformation;

[0017] Restore the dimension of the feature map tensor of the grouped convolution operation;

[0018] The feature map tensor obtained by dimension restoration is convolved and the number of channels is restored.

[0019] In the embodiment of the present invention, the grouped convolution is a 1×1 grouped convolution.

[0020] A first object of the present invention is to provide a feature map upsampling system based on tensor dimension transformation and convolution, the system comprising:

[0021] The input module is used to input feature maps;

[0022] The convolution operation module is used to perform convolution operations on the tensor of the feature map;

[0023] The dimension and convolution module is used to perform dimension transformation and convolution on the feature map tensor obtained by the convolution operation;

[0024] The output module is used to output the upsampling results obtained by dimension transformation and convolution.

[0025] In an embodiment of the present invention, the dimension and grouped convolution module includes a dimension submodule and a grouped convolution submodule;

[0026] The dimension submodule is used to perform dimension transformation on the feature map tensor obtained by the convolution operation, and to perform dimension restoration on the feature map tensor of the grouped convolution operation;

[0027] The grouped convolution submodule is used to perform grouped convolution operations on the feature map tensor obtained by dimensional transformation, and to perform convolution and channel number restoration on the feature map tensor obtained by dimensional restoration.

[0028] A third object of the present invention is to provide an electronic device, the device comprising: a processor, the processor being coupled to a memory;

[0029] The memory is used to store computer programs;

[0030] The processor is used to execute the computer program stored in the memory so that the electronic device executes the method as described above.

[0031] A fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and when the program or instruction is executed on a computer, the computer executes the method as described above.

[0032] Beneficial effects of the present invention:

[0033] The present invention provides a method and system for upsampling a feature map based on tensor dimension transformation and convolution, which realizes upsampling through convolution operation and dimension operation, and is mainly suitable for the upsampling process of small and medium-scale feature maps. The calculation process is simpler than the traditional method, the amount of calculation is lower than the transposed convolution, the calculation accuracy is higher, and adaptive upsampling of data can be achieved through learning. The method can be built into the encoder of the convolutional neural network image segmentation model to perform upsampling operations on feature maps, and can also be trained by deep learning methods to achieve amplification and generation of specific image data.

[0034] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a feature map upsampling method based on tensor dimension transformation and convolution according to an embodiment of the present invention is shown;

[0037] Figure 2 shows a labeled training sample graph according to an embodiment of the present invention;

[0038] Figure 3 shows a test sample diagram after annotation according to an embodiment of the present invention;

[0039] Figure 4 A certain A channel diagram of input according to an embodiment of the present invention is shown;

[0040] Figure 5 A certain A channel diagram of the sampling result outputted by the up-sampling method provided by the embodiment of the present invention is shown according to the embodiment of the present invention;

[0041] Figure 6 A framework diagram of a feature map upsampling system based on tensor dimension transformation and convolution according to an embodiment of the present invention is shown;

[0042] Figure 7 A framework diagram of an electronic device according to an embodiment of the present invention is shown;

[0043] In the figure:

[0044] Input module 1; convolution operation module 2; dimension and convolution module 3; output module 4; electronic device 300; processor 301; memory 302. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, a feature map upsampling method based on tensor dimension transformation and convolution according to an embodiment of the present invention includes:

[0047] Step S1, input feature map;

[0048] Step S2, performing a convolution operation on the tensor of the feature map so that the number of its channels becomes a square number;

[0049] Step S3, performing dimension transformation and convolution on the feature map tensor obtained by the convolution operation;

[0050] Step S4: output the upsampling result obtained by dimensional transformation and convolution.

[0051] In step S1, let the tensor of the feature map be X(C 0 ,H 0 ,W 0 ), where C 0 is the number of channels of X, H and W are the height and width of X respectively;

[0052] Specifically, in step S2, the convolution operation is performed on the tensor of the feature map so that the number of its channels becomes a square number, that is:

[0053] Perform a 1×1 convolution operation on X to transform its channel number to the nearest neighbor square number C 1 , get the tensor X 1 , specifically as shown in formula (1):

[0054]

[0055] In formula (1), Conv 1×1 () represents 1×1 convolution.

[0056] In step S3, the dimension transformation and convolution of the feature map tensor obtained by the convolution operation include:

[0057] Step A1: transform the dimension of the feature map tensor obtained by the convolution operation, transforming the spatial dimension into the channel dimension;

[0058] Step A2: performing group convolution operation on the feature map tensor obtained by dimensional transformation to increase the number of its channels to several times of the original number;

[0059] Step A3: restore the dimension of the feature map tensor of the grouped convolution operation, and restore the spatial dimension from the channel direction;

[0060] Step A4: perform convolution and channel number restoration on the feature map tensor obtained by dimension restoration.

[0061] In step A1, the feature map tensor obtained by the convolution operation is transformed in dimension, that is:

[0062] X 1 To perform dimensional transformation, first expand the pixels in the spatial direction and transform them to the channel direction; then transform the pixels in the channel direction to the spatial direction in a square arrangement to obtain X 1 ', whose shape is

[0063] In step A2, the feature map tensor obtained by dimensional transformation is subjected to group convolution operation, namely:

[0064] X 1 'Perform 1×1 group convolution to expand the number of channels to the original n 2 times, get X 2 Tensor, as shown in formula (2):

[0065]

[0066] In formula (2), G is the number of convolution groups.

[0067] In step A3, the dimension of the feature map tensor of the grouped convolution operation is restored, that is:

[0068] X 2 Perform dimension reduction, that is, perform the inverse operation of step A1, and obtain X after reduction. 2 ', its shape is (C 1 ,nH 0 ,nW 0 ).

[0069] In step A4, the feature map tensor obtained by dimension reduction is convolved and the number of channels is reduced, that is:

[0070] X 2 'Perform 1×1 convolution to fuse the features and restore the number of channels to C 0 Or output channel value C out , and obtain the upsampling result Y, as shown in formula (3).

[0071] Y(C0 (or C out ),nH 0 ,nW 0 )=Conv 1×1 (X' 2 (C 1 ,nH 0 ,nW 0 )) (3)

[0072] In step S4, the up-sampling result obtained by the output dimension transformation and convolution, that is, the up-sampling result Y is output.

[0073] To more clearly illustrate a feature map upsampling method based on tensor dimension transformation and convolution of the present invention, the following is an example of a specific picture processing process:

[0074] First, the feature map in the embodiment of the present invention is obtained by processing the original image input into the CNN backbone segmentation model. For example, the embodiment of the present invention builds a CNN image model with a codec structure. The encoder is composed of 4 convolutional feature extractors. Each feature extractor is composed of 2 3×3 convolutional layers and 1 downsampling maximum pooling layer. The decoder has the same structure as the encoder. The main difference is that the maximum pooling layer is replaced by the upsampling layer described in the present invention, the output layer uses bilinear interpolation upsampling, and the activation function is the softmax function.

[0075] The network input is a 224×224×3 color image. After four feature extraction layers, the resolution of the output feature map is 224→112→56→28→14, and the number of channels is 3→128→256→512→512. The decoder input feature map resolution is 14, the number of channels is 512, and the output feature map resolution is 28→56→112→224, and the number of channels is 512→256→128→3.

[0076] Taking the second decoding module of the decoder as an example, the input image tensor is X(512, 28, 28), and the upsampling method provided in the embodiment of the present invention is used for processing:

[0077] Step 1: Perform a 1×1 convolution operation on X to transform its channel number to the nearest neighbor square number C 1 , get the tensor X 1 , as shown in formula (4):

[0078]

[0079] In formula (4), Conv 1×1 () represents 1×1 convolution, and round() is the nearest neighbor rounding function.

[0080] Step 2: X 1 To perform dimensional transformation, first expand the pixels in the spatial direction and transform them to the channel direction; then transform the pixels in the channel direction to the spatial direction in a square arrangement to obtain X 1 ', whose shape is (784,23,23).

[0081] Step 3: X 1 'Perform 1×1 group convolution to expand the number of channels to the original 2 2 times, get X 2 Tensor, as shown in formula (5):

[0082] X 2 (4 784,23,23) = Conv 1×1 (X 1 (784,23,23), g = 784) (5)

[0083] In formula (5), G is the number of convolution groups.

[0084] Step 4: X 2 Perform dimension reduction, that is, perform the inverse operation of step 2, and obtain X after reduction 2 ', its shape is (529,2×28,2×28)=(529,56,56).

[0085] Step 5: X 2 'Perform 1×1 convolution to change its channel number to output channel C out =256, and the upsampling result Y is obtained, which is specifically shown in formula (6):

[0086] Y(256,56,56)=Conv 1×1 (X' 2 (529,56,56)) (6)

[0087] And output the up-sampling result Y obtained above;

[0088] The CNN image segmentation model equipped with the upsampling algorithm of the present invention is trained. The dataset used is a steel microstructure phase content dataset, which is the semantic segmentation data of the F phase and the P phase in the ferrite (F) + pearlite (P) structure. The labeled training samples are as follows: Figure 2 The model was trained with momentum optimizer and learning rate 0.001 for 100 cycles and then stopped.

[0089] After the training is completed, the test samples are input into the model. The test samples are as follows: Figure 3 Check the A channel of the upsampling algorithm input in the second decoding module of the decoder as shown in FIG. Figure 4As shown, the upsampling algorithm of the present invention is used to obtain a sampling result diagram (corresponding to a certain A channel), such as Figure 5 shown.

[0090] like Figure 6 As shown, an embodiment of the present invention provides a feature map upsampling system based on tensor dimension transformation and convolution, and the system includes:

[0091] Input module 1 is used to input feature maps;

[0092] The convolution operation module 2 is used to perform convolution operation on the tensor of the feature map;

[0093] The dimension convolution module 3 is used to perform dimension transformation and convolution on the feature map tensor obtained by the convolution operation;

[0094] The output module 4 is used to output the upsampling result obtained by dimensionality transformation and convolution.

[0095] In an embodiment of the present invention, the dimension and convolution module 3 includes a dimension submodule and a grouped convolution submodule;

[0096] The dimension submodule is used to perform dimension transformation on the feature map tensor obtained by the convolution operation, and to perform dimension restoration on the feature map tensor of the grouped convolution operation;

[0097] The grouped convolution submodule is used to perform grouped convolution operations on the feature map tensor obtained by dimensional transformation, and to perform convolution and channel number restoration on the feature map tensor obtained by dimensional restoration.

[0098] like Figure 7 As shown, some embodiments of the present invention provide an electronic device, the electronic device 300, comprising: a processor 301, the processor 301 is coupled to a memory 302;

[0099] The memory 302 is used to store computer programs;

[0100] The processor 301 is configured to execute the computer program stored in the memory 302 so that the electronic device executes the method described in the above embodiment.

[0101] In certain embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program or instruction. When the program or instruction is executed on a computer, the computer executes the method described in the above embodiments.

[0102] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an electronic device or a device.

[0103] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A feature map upsampling method based on tensor dimension transformation and convolution, characterized in that: include: Input feature map; Perform convolution operation on the tensor of feature map; Perform dimension transformation and convolution on the feature map tensor obtained by the convolution operation; Output the upsampled result of dimension transformation and convolution.

2. The feature map upsampling method based on tensor dimension transformation and convolution according to claim 1, characterized in that: The convolution operation is a 1×1 convolution.

3. The feature map upsampling method based on tensor dimension transformation and convolution according to claim 1 or 2, characterized in that: The step of performing dimension transformation and convolution on the feature map tensor obtained by the convolution operation includes: Transform the dimension of the feature map tensor obtained by the convolution operation; Perform group convolution operation on the feature map tensor obtained by dimension transformation; Restore the dimension of the feature map tensor of the grouped convolution operation; The feature map tensor obtained by dimension restoration is convolved and the number of channels is restored.

4. The feature map upsampling method based on tensor dimension transformation and convolution according to claim 3, characterized in that: The dimension transformation is carried out in such a way that the spatial dimension and the channel dimension are interchanged in direction.

5. The feature map upsampling method based on tensor dimension transformation and convolution according to claim 3, characterized in that: The grouped convolution is a 1×1 grouped convolution.

6. A feature map upsampling system based on tensor dimension transformation and convolution, characterized in that: include: The input module is used to input feature maps; The convolution operation module is used to perform convolution operations on the tensor of the feature map; The dimension and convolution module is used to perform dimension transformation and convolution on the feature map tensor obtained by the convolution operation; The output module is used to output the upsampling results obtained by dimension transformation and convolution.

7. The feature map upsampling system based on tensor dimension transformation and convolution according to claim 6, characterized in that: The dimensional and grouped convolution module includes a dimensional submodule and a grouped convolution submodule; The dimension submodule is used to perform dimension transformation on the feature map tensor obtained by the convolution operation, and to perform dimension restoration on the feature map tensor of the grouped convolution operation; The grouped convolution submodule is used to perform grouped convolution operations on the feature map tensor obtained by dimensional transformation, and to perform convolution and channel number restoration on the feature map tensor obtained by dimensional restoration.

8. The feature map upsampling system based on tensor dimension transformation and convolution according to any one of claims 6-7, characterized in that: The convolution operation is a 1×1 convolution.

9. An electronic device, characterized in that: include: a processor coupled to the memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, and when the program or the instruction is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images