A non-local convolution modeling method and device based on feature block folding

Abstract

The present invention belongs to the field of image recognition technology, and specifically relates to a non-local convolution modeling method and device based on feature block folding. The method comprises: constructing an FFNL module, the FFNL module comprising an FF submodule, an RFF submodule and an NL submodule; embedding the FFNL module into a shallow stage of a common convolutional network to obtain a non-local convolution classification model based on feature block folding; acquiring training data and using the training data to train the non-local convolution classification model based on feature block folding to obtain a trained non-local convolution classification model based on feature block folding; using the trained non-local convolution classification model based on feature block folding to process data to be classified to obtain a classification result. The present invention can help a convolutional neural network establish a non-local feature learning mechanism based on feature blocks, thereby obtaining a larger effective receptive field and better performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a non-local convolution modeling method and device based on feature block folding. Background Art

[0002] With the rapid development of computer technology, information processing technology, and visual communication technology, humanity has entered a new information age. Convolutional neural networks continue to push the boundaries of deep learning in the field of image processing. However, convolutional neural networks also have some challenges, among which how to increase the effective receptive field size of convolutional networks is a key issue.

[0003] Due to the non-local connectivity mechanism in convolution operations, convolutional neural networks cannot directly learn non-local features. Traditionally, convolutional neural networks (CNNs) learn low-level features in shallow layers and generalize these features into global semantic information in deeper layers. However, this process has certain flaws: CNNs cannot capture cross-distance dependencies based on feature patches. Visual information differs significantly from natural language. Natural language, a highly abstract form of information for humans, has less redundancy, with closer words often being more correlated. Image information, on the other hand, is sparse and redundant. This sparse redundancy is explained by the non-local self-similarity prior of images, which states that a large amount of cross-distance information based on patches in an image is similar. Furthermore, from a feature learning perspective, it is necessary to assume that the large amount of cross-distance patch information in an image requires feature connections, not just similarity. Establishing non-local learning based on feature patches not only improves the model's receptive field but also has the potential to increase feature diversity and reduce feature redundancy within channels. However, convolutional neural networks lack non-local learning based on feature blocks. People usually only emphasize long-distance dependencies and use feature point-based modeling methods such as non-local and dilated convolutions. However, these methods are not based on feature blocks and have high computational costs or little effect. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a non-local convolution modeling method and device based on feature block folding, which includes:

[0005] S1: Construct the FFNL module, which includes the FF submodule, RFF submodule and NL submodule;

[0006] S2: The FFNL module is embedded into the shallow stage of the ordinary convolutional network to obtain a non-local convolutional classification model based on feature block folding. The non-local convolutional classification model based on feature block folding includes the front-end network, the FFNL module and the back-end network.

[0007] S3: Acquire training data and use the training data to train a non-local convolution classification model based on feature block folding, to obtain a trained non-local convolution classification model based on feature block folding;

[0008] S4: Use the trained non-local convolution classification model based on feature block folding to process the classification data and obtain the classification results.

[0009] Preferably, the ordinary convolutional network is a ResNet network, which includes 4 backbone modules, and the FFNL module is embedded between the first backbone module and the second backbone module of the ResNet network.

[0010] Preferably, the process of processing the data to be classified by the non-local convolution classification model based on feature block folding includes:

[0011] S41: The front-end network of the non-local convolution classification model based on feature block folding extracts features from the data to be classified and obtains primary features;

[0012] S42: Use the FFNL module to extract the primary features and obtain secondary features;

[0013] S43: The secondary features are processed using the back-end network of the non-local convolution classification model based on feature block folding to obtain the classification results.

[0014] Furthermore, the process of extracting primary features using the FFNL module includes:

[0015] The FF submodule is used to divide the primary features into multiple feature blocks in the spatial dimension; multiple feature blocks are folded in the channel dimension to obtain folded features;

[0016] The NL submodule is used to perform feature learning on the folded features to obtain the first intermediate features;

[0017] The RFF submodule is used to restore all feature blocks of the first intermediate feature to the same spatial position as the primary feature to obtain the second intermediate feature;

[0018] The primary feature is added to the second intermediate feature to obtain the secondary feature.

[0019] Furthermore, the rule for folding multiple feature blocks in the channel dimension is: sorting the feature blocks according to the sorting priorities of the horizontal spatial dimension, the vertical spatial dimension, and the channel dimension and folding them according to the sorting.

[0020] Furthermore, the process of processing secondary features by the back-end network of the non-local convolution classification model based on feature block folding includes: the back-end network includes a pooling layer and a fully connected layer; the secondary features are output to the pooling layer for average pooling to obtain a one-dimensional feature vector; the one-dimensional feature vector is input to the fully connected layer to obtain the classification result.

[0021] Preferably, the NL submodule includes a first convolution layer, a second convolution layer, and a third convolution layer.

[0022] Furthermore, the convolution kernel sizes of the first, second, and third convolution layers are 3×3, 3×3, and 1×1, respectively, and the number of convolution kernel groups is equal to the channel dimension of the input features of the FFNL module.

[0023] A non-local convolution modeling device based on feature block folding, comprising: an acquisition unit, a model building unit, a training unit and an output unit;

[0024] The acquisition unit is used to acquire training data;

[0025] The model construction unit is used to construct the FFNL module and embed the FFNL module into the shallow stage of the ordinary convolutional network to obtain a non-local convolution classification model based on feature block folding;

[0026] The training unit is used to train the non-local convolution classification model based on feature block folding using training data, save model parameters, and obtain a trained non-local convolution classification model based on feature block folding;

[0027] The output unit is used to output the trained non-local convolution classification model based on feature block folding.

[0028] The beneficial effects of the present invention are as follows: the present invention discovers the existence of non-local priors in the feature maps in the shallow stage of the network, and captures these priors by designing the FFNL module, thereby realizing feature learning based on the internal and inter-feature blocks, and embedding the FFNL module into the shallow stage of the convolutional neural network. At a very low additional cost, it helps the convolutional neural network establish a non-local feature learning mechanism based on feature blocks, thereby obtaining a larger effective receptive field and better performance, and to a certain extent solves the problem that the convolutional neural network cannot capture cross-distance dependencies based on feature blocks, thereby effectively improving the classification accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the FFNL module structure in the present invention;

[0030] Figure 2 This is a schematic diagram of the ResNet network embedded in the FFNL module of the present invention;

[0031] Figure 3 This is a schematic diagram of the FF submodule in the present invention;

[0032] Figure 4 Schematic diagram of the NL submodule in the present invention;

[0033] Figure 5 Schematic diagram of the RFF submodule in the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] The present invention proposes a non-local convolution modeling method and device based on feature block folding, the method comprising:

[0036] S1: Construct the FFNL module, which includes the FF submodule, RFF submodule and NL submodule.

[0037] like Figure 1 As shown in the figure, the present invention designs a convolutional feature learning module based on feature block folding and embedded non-locality, referred to as the FFLN module, which includes three parts: feature block folding submodule (feature fold, FF), feature block reverse folding submodule (reverse feature fold, RFF), and non-local convolution module (non-local, NL).

[0038] S2: The FFNL module is embedded into the shallow stage of the ordinary convolutional network to obtain a non-local convolutional classification model based on feature block folding; wherein, the non-local convolutional classification model based on feature block folding includes the front-end network, the FFNL module and the back-end network.

[0039] The FFNL module is mainly used to capture the non-local priors in the feature map. According to the feature map visualization, it can be seen that the FFNL module is not suitable for direct embedding in the deep stage of the network. Since the non-local priors of the input image gradually disappear in the network with continuous nonlinear transformations, capturing these non-local priors is exactly the inductive bias that the FFNL module wants to achieve. If used in the deep stage of the network, it will have a negative effect. And after experimental verification, the FFNL module is most suitable for embedding in the shallow stage of the network. For example, Figure 2As shown in the figure, if the ordinary convolutional network is a ResNet network (residual neural network), and the ResNet network has 4 backbone modules, then the FFNL module will be embedded between the first backbone module and the second backbone module of the ResNet network.

[0040] S3: Acquire training data and use the training data to train a non-local convolution classification model based on feature block folding to obtain a trained non-local convolution classification model based on feature block folding.

[0041] The training data can be a labeled image training set. During model training, the image training set is input, the image classification results are output, and the loss is calculated with the true labels. Backpropagation is then performed to adjust the loss function and image feature extraction. Model training is complete when the loss function is minimized or the maximum number of iterations is reached. The current network parameters are saved, resulting in a trained non-local convolutional classification model based on feature block folding. To observe the model's effective receptive field, the last convolutional feature layer is extracted to calculate the backpropagation average gradient of its central feature point at the network input feature layer. By visualizing these gradients, the model's effective receptive field can be observed.

[0042] S4: Use the trained non-local convolution classification model based on feature block folding to process the classification data and obtain the classification results.

[0043] In practical applications, the data to be classified may be image data to be classified. The image data to be classified is input into a trained non-local convolution classification model based on feature block folding to obtain an image classification result.

[0044] In some embodiments, the input data to be classified is an image to be classified, and the process of processing the image to be classified by the non-local convolution classification model based on feature block folding includes:

[0045] S41: The front-end network of the non-local convolution classification model based on feature block folding extracts features from the image to be classified to obtain primary features.

[0046] S42: Use the FFNL module to extract the primary features and obtain secondary features. The specific process is as follows:

[0047] Assume that the feature dimension of the input feature of the FFNL module is<N,C,H,W> , where N is the batch dimension, H is the horizontal spatial dimension, W is the vertical spatial dimension, and C is the channel dimension.

[0048] like Figure 3As shown in the figure, the FF submodule is used to divide the primary features into multiple feature blocks in the spatial dimension; multiple feature blocks are folded in the channel dimension to obtain folded features; specifically: the input feature dimension is<N,C,H,W> ; Divide the input features into feature blocks in both spatial and channel dimensions. If the H spatial dimension is divided twice, the W spatial dimension is divided twice, and the channel dimension is divided C times, then there are N×2×2×C feature blocks, and the size of each feature block is<H / 2,W / 2,> , these feature blocks are folded into the channel dimension and reconstructed into new features. The rule for folding multiple feature blocks in the channel dimension is: sort the feature blocks according to the sorting priority of the horizontal spatial dimension, the vertical spatial dimension, and the channel dimension and fold them according to the sorting. The dimension of the folded feature is<N,2×2×C,H / 2,W / 2,> .

[0049] like Figure 4 As shown in the figure, the NL submodule is used to learn the folded features to obtain the first intermediate features; the input of the NL submodule is the output of the FF submodule. The NL submodule consists of three layers of convolution, namely the first layer of convolution, the second layer of convolution and the third layer of convolution. The number of groups of the convolution kernel is the channel dimension of the input feature of the FFNL module. Among them, the parameters of the first layer of convolution are the convolution kernel size of 3×3, the number of groups is C, the number of input channels is 2×2×C, and the number of output channels is 2×2×C; the parameters of the second layer of convolution are the same as those of the first layer; the parameters of the third layer of convolution are the convolution kernel size of 1×1, the number of groups is C, the number of input channels is 2×2×C, and the number of output channels is 2×2×C; at this time, the dimension of the output feature of the NL submodule is<N,2×2×C,H / 2,W / 2,> .

[0050] like Figure 5 As shown in the figure, the RFF submodule is used to restore all the feature blocks of the first intermediate feature to the same spatial position as the primary feature to obtain the second intermediate feature. The RFF submodule is a completely reversible operation of the FF submodule. Reversibility means that all feature blocks are eventually relocated to their original spatial positions, and the original positional relationship between them is restored, that is, all feature blocks are restored to the same spatial position as the primary feature. The reversible operation of this step is very important because the residual learning structure will be added later. If the original feature block is not relocated to its proper position, it will not only cause the residual learning structure to fail, but also the entire FFNL module will have a negative effect.

[0051] Next, the primary features are added to the second intermediate features to obtain the secondary features. Because the NL convolution submodule in the FFNL module cannot guarantee the establishment of effective cross-region feature connections, excessively strong inductive bias will cause the convolutional neural network to fall into a suboptimal solution. Therefore, by adding the primary features to the second intermediate features to construct a residual learning structure, this problem can be avoided.

[0052] S43: The secondary features are processed using the back-end network of the non-local convolution classification model based on feature block folding to obtain the image classification results.

[0053] The back-end network includes a pooling layer and a fully connected layer; the secondary features are output to the pooling layer for average pooling to obtain a one-dimensional feature vector; the one-dimensional feature vector is input to the fully connected layer to obtain the image classification result; preferably, other network structures for feature extraction can also be included before the pooling layer.

[0054] The present invention also proposes a non-local convolution modeling device based on feature block folding, which can be used to implement the above-mentioned non-local convolution modeling method based on feature block folding, including: an acquisition unit, a model building unit, a training unit and an output unit;

[0055] The acquisition unit is used to acquire training data;

[0056] The model construction unit is used to construct the FFNL module and embed the FFNL module into the shallow stage of the ordinary convolutional network to obtain a non-local convolution classification model based on feature block folding;

[0057] The training unit is used to train the non-local convolution classification model based on feature block folding using training data, save model parameters, and obtain a trained non-local convolution classification model based on feature block folding;

[0058] The output unit is used to output the trained non-local convolution classification model based on feature block folding.

[0059] Obtain the image to be classified, and input the image to be classified into the trained image classification model based on feature block folding. The model performs the above processing on the image to be classified to obtain the image classification result.

[0060] In the FFNL module, the FF feature block folding sub-model is not learnable and only includes the computational cost of feature space deformation. The NL convolution sub-module is learnable, but its parameter count and computational overhead are extremely low, primarily due to the use of convolution with a large number of groups. Based on the calculation method of grouped convolution, the parameter count and computational overhead of the NL convolution sub-module are approximately 1 / C times that of a similar convolution module with a grouping of 1, where C is the channel dimension of the input features of the FFNL module. Of course, considering the underlying implementation of grouped convolution, its computational speed will be slightly slower than theoretical, but the actual number of parameters and computational cost are still extremely low.

[0061] In summary, the FFNL module designed by the present invention divides the input features into multiple feature blocks in the spatial dimension, and folds these feature blocks in the channel dimension. At this time, feature learning is performed using convolution of specific groups, that is, feature connections are established between all feature blocks from the same feature map, which is equivalent to realizing feature learning based on feature blocks within the feature map. When the input features enter the FFNL module, the input features will be sent to the FF submodule, NL submodule, and RFF submodule for processing in turn. The new features obtained after processing establish non-local feature learning, but need to be added to the input features again to construct residual learning to prevent the FFNL module from generating too strong inductive bias; by performing specific deformation processing and convolution learning on the input features to capture the inductive bias, the present invention helps the convolutional neural network establish a non-local feature learning mechanism based on feature blocks at a very low additional cost, thereby obtaining a larger effective receptive field and better performance.

[0062] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A non-local convolution modeling method based on feature block folding, characterized in that: include: S1: Construct the FFNL module, which includes the FF submodule, RFF submodule and NL submodule; S2: The FFNL module is embedded into the shallow stage of the ordinary convolutional network to obtain a non-local convolutional classification model based on feature block folding. The non-local convolutional classification model based on feature block folding includes the front-end network, the FFNL module and the back-end network. S3: Acquire training data and use the training data to train a non-local convolution classification model based on feature block folding, to obtain a trained non-local convolution classification model based on feature block folding; S4: Using the trained non-local convolution classification model based on feature block folding to process the classification data to obtain a classification result; the process of using the non-local convolution classification model based on feature block folding to process the classification data includes: S41: The front-end network of the non-local convolution classification model based on feature block folding extracts features from the data to be classified and obtains primary features; S42: Using the FFNL module to extract the primary features to obtain secondary features. The process of using the FFNL module to extract the primary features includes: The FF submodule is used to divide the primary features into multiple feature blocks in the spatial dimension; the multiple feature blocks are folded in the channel dimension to obtain folded features; the rule for folding multiple feature blocks in the channel dimension is: the feature blocks are sorted according to the sorting priority of the horizontal spatial dimension, the vertical spatial dimension, and the channel dimension, and folded according to the sorting; The NL submodule is used to perform feature learning on the folded features to obtain the first intermediate features; The RFF submodule is used to restore all feature blocks of the first intermediate feature to the same spatial position as the primary feature to obtain the second intermediate feature; Add the primary feature to the second intermediate feature to obtain the secondary feature; S43: The secondary features are processed using the back-end network of the non-local convolution classification model based on feature block folding to obtain the classification results.

2. The non-local convolution modeling method based on feature block folding according to claim 1, characterized in that: The ordinary convolutional network is a ResNet network, which includes four backbone modules, and the FFNL module is embedded between the first backbone module and the second backbone module of the ResNet network.

3. The non-local convolution modeling method based on feature block folding according to claim 1, characterized in that: The process of processing secondary features by the back-end network of the non-local convolution classification model based on feature block folding includes: the back-end network includes a pooling layer and a fully connected layer; the secondary features are output to the pooling layer for average pooling to obtain a one-dimensional feature vector; the one-dimensional feature vector is input to the fully connected layer to obtain the classification result.

4. The non-local convolution modeling method based on feature block folding according to claim 1, characterized in that: The NL submodule includes a first convolution layer, a second convolution layer, and a third convolution layer.

5. The non-local convolution modeling method based on feature block folding according to claim 4, characterized in that: The convolution kernel sizes of the first, second, and third convolution layers are 3×3, 3×3, and 1×1, respectively, and the number of convolution kernel groups is equal to the channel dimension of the input features of the FFNL module.

6. A non-local convolution modeling device based on feature block folding, the device being used to execute any one of the non-local convolution modeling methods based on feature block folding according to claims 1 to 5, characterized in that: include: Acquisition unit, model building unit, training unit and output unit; The acquisition unit is used to acquire training data; The model construction unit is used to construct the FFNL module and embed the FFNL module into the shallow stage of the ordinary convolutional network to obtain a non-local convolution classification model based on feature block folding; The training unit is used to train the non-local convolution classification model based on feature block folding using training data, save model parameters, and obtain a trained non-local convolution classification model based on feature block folding; The output unit is used to output the trained non-local convolution classification model based on feature block folding.

Images