A method for image classification using a feature mixing network model
By using a feature-mixing network model and channel splitting and CSPNEXt mixer to process high and low frequency signals, the problem of simultaneously processing high and low frequency information in existing technologies is solved, thus improving the accuracy and efficiency of image classification.
Patent Information
- Application Number
- CN202211558094.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-06
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2042-12-06
AI Technical Summary
Existing image classification methods struggle to effectively process both high-frequency and low-frequency information features simultaneously, resulting in limited model performance.
A feature mixing network model is adopted. Shallow features are extracted through 1×1 convolutional layers and the number of channels is increased. The features are divided into high-frequency and low-frequency signals using a channel splitting mechanism. These signals are then input into the CSPNEXt mixer for processing. The high-frequency mixer uses depthwise separable convolution and linear layers, while the low-frequency mixer uses dilated convolution and average pooling. An upsampling operation is also performed, and the final result is obtained by concatenation and fusion.
It improves the performance of image classification, balances the processing capabilities of high and low frequency signals, and achieves better accuracy and less computation, outperforming traditional Transformer and CNN models.
Smart Images
Figure CN115861696B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image classification, and particularly relates to a method for image classification using a feature mixing network model. BACKGROUND
[0002] Convolutional neural networks (CNNs) have long been the mainstream of computer vision modeling. Since AlexNet first used convolutional neural networks for image classification tasks, it has achieved great success. CNN architecture is developing towards more complex, larger scale and more extensive connection of convolutional form. CNN has become the actual backbone network of various computer vision tasks, and has promoted the unprecedented development of the entire field.
[0003] CNN is the fact model of computer vision tasks because they can model local dependencies and extract high-frequency information. Zhou Zhengming is introducing CNN in serial or parallel mode into the Transformer model. For example, Dai Zihang et al. vertically stack convolutional layers and attention layers in a principled way. It is also pointed out in Xia Xin et al. TRT-ViT that it is impractical to deploy only Transformer models on limited resource devices, and outlines a set of principles for designing models that are as good as Transformer models in performance and as fast as ResNet in prediction. However, grouping convolution and attention in some way means that high and low frequency components cannot be processed at the same time.
[0004] Starting from the MLP Mixer, models based on the MLP architecture achieve competitive performance on the ImageNet dataset by completely replacing the self-attention-based mechanism of ViT with MLP. This research direction makes us question the use of self-attention mechanisms. And the attention mechanism based on the Transformer has a quadratic complexity with the width and height of the input picture. Some studies show that convolutional layers usually have better generalization ability and faster convergence speed because they have strong inductive bias prior knowledge. Alexey Dosovitskiy et al. proposed in the paper An image is worth 16x16 words: Transformers for image recognition at scale that VisionTransformer (ViT) uses a series of embedded image blocks as the input of the standard converter, which is the first non-convolutional converter that performs as well as CNN models. However, ViT requires very large datasets such as ImageNet21K and JFT300M for training, and lacks the inductive bias that is unique to CNN. Li Yuan, Chen Yunpeng et al. proposed in the paper Tokens-to-token vit: Training vision transformers from scratch on imagenet that T2T-ViT introduces a hierarchical Tokens-to-Token (T2T) conversion to encode the important local structure of each token instead of the ordinary tokenization used in ViT. Unlike these methods, we propose a dual-path architecture that focuses on both high-frequency and low-frequency signal features, while ViT only focuses on global features and T2T-ViT only focuses on local features. Our research shows that deep separable convolution can be effectively incorporated into layers through a simple feature mixer. SUMMARY
[0005] The present application aims at the deficiency that the prior art image classification method cannot well consider the low-frequency information and high-frequency information features at the same time, and provides a method for image classification using a feature mixing network model. The application can extract high-frequency and low-frequency signal features, and use a new channel splitting mechanism to enable the model to have different capabilities for high and low frequency signals at different layers.
[0006] To solve this technical problem, the technical solution of the present application is: a method for image classification using a feature mixing network model,
[0007] Step one, given an image I as input, use a 1x1 convolutional layer and an activation function to extract shallow features H0, and expand the channel number of the feature map to twice the original.
[0008] H0 = F1(Bn(I))
[0009] Wherein F1 represents a 1x1 convolution operation, and Bn represents an activation function.
[0010] Step two, the extracted features are divided into X h and X l in the channel dimension using a channel splitting mechanism.
[0011] X h ,X h = Split(H0)
[0012] Wherein Split represents a channel splitting mechanism.
[0013] Step three, X h and X l are input into a plurality of CSPNeXt mixer feature mixers, respectively, each CSPNeXtmixer obtains output features Y h and Y l , and each CSPNeXt mixer is composed of a low-frequency mixer and a high-frequency mixer, wherein the high-frequency mixer is composed of a linear layer (Linear) and a depth separable convolution (DwConv), and the low-frequency mixer is composed of a dilated convolution (Dilated Convolution), an average pooling (AvePool) and an up sampling (Upsample):
[0014] Y h = DwConv(Linear(X h ))
[0015] Y1 = Upsample(AvePooling(DilateConv(X l )))
[0016] Step four, the output features Y h and Y l obtained in step three are concatenated in the feature map dimension direction by a concatenation operation to obtain the feature Y c :
[0017] Y c = Concat(Y l ,Y h )
[0018] Wherein, Concat is a concatenation operation, and the channel dimension is restored to before the splitting operation by the concatenation operation.
[0019] Step five, a 1x1 convolution is adopted to the concatenated high and low frequency features Y cThe features are fused again effectively. The features H1 after feature fusion are input to the classification head to obtain a final classification result;
[0020] A = ClsHead(H1)
[0021] wherein A represents the final classification result, and ClsHead represents the classification head;
[0022] Preferably, the CSPNeXt mixer module comprises a low-frequency mixer and a high-frequency mixer. A feature mixing network model suitable for image classification is composed of a plurality of CSPNeXt mixer modules and a channel splitting mechanism. The features are divided by the splitting mechanism, and then the global and local features are interactively fused by the CSPNeXt mixer.
[0023] Further preferably, the CSPNeXt mixer module works as follows:
[0024] The input of the CSPNeXt mixer module is split into X h and X l
[0025] X h is input to the high-frequency mixer to process the high-frequency signals in the features. First, a linear projection is performed using Linear, and then a DWConv with a kernel size of 7x7 is used. The large kernel convolution of 7x7 is beneficial to expand the receptive field and can effectively extract high-frequency signals such as edges and angles. At the same time, the input feature X l is input to the low-frequency mixer. After DilatedConvolution, the global receptive field is obtained, and then global feature interaction is performed through global average pooling (Ave Pool) while reducing the width and height of the feature map. Finally, the feature map is restored to its original size through the up-sampling layer (Upsample).
[0026] Y h = DwConv(Linear(X h ))
[0027] Y1 = Upsample(AvePooling(DilateConv(X l )))
[0028] Further preferably, the channel splitting mechanism is that before the features are input to the CSPNeXt mixer, the features are divided according to the channel dimension. More channel dimensions are divided for the high-frequency mixer at the shallow layer of the model, and more channels are divided for the low-frequency mixer at the deep layer of the model. Different layers exhibit different capture capabilities for high and low frequency signals.
[0029] Preferably, random cropping, horizontal flipping, and vertical flipping are used to expand the training data.
[0030] Preferably, the multi-channel aggregation network is implemented on the PyTorch platform and tested with an Nvidia A100 graphics card.
[0031] Preferably, the activation function is a LeakyReLU function or a Relu function.
[0032] Technical solutions, the beneficial effects of the present application are:
[0033] The present application proposes a method for image classification using a feature mixing network model, which improves the performance of the network model classification while maintaining high efficiency. The CSPNeXt mixer of the present application can extract high-frequency and low-frequency signal (local and global) features, and use a new channel splitting mechanism to enable the model to have different capabilities for high and low frequency signal processing at different layers.
[0034] A large number of experiments on multiple benchmark datasets show that the proposed network has better accuracy and fewer parameter quantities compared with some latest methods based on Transformer; the method has superiority in accuracy and computational amount.
[0035] Thus, the above-mentioned purposes of the present application are achieved. BRIEF DESCRIPTION OF DRAWINGS
[0036] Figure 1 is the overall architecture flowchart of the feature mixing network model according to the present application;
[0037] Figure 2 is the CSPNeXt mixer module diagram and channel splitting mechanism diagram in the present application;
[0038] Figure 3 is the feature mixing model and Swin Transformer comparison heat map in the present application;
[0039] Figure 4 The feature visualization diagram of each layer of the feature mixing model in the present application is provided. DETAILED DESCRIPTION
[0040] In order to further explain the technical solutions of the present application, the present application will be described in detail below through specific embodiments.
[0041] The present embodiment discloses a method for image classification using a feature mixing network model, and the specific process is as shown in Figure 1 , including the following steps:
[0042] Step one, given an image I as input, use 1x1 convolution layer and activation function to extract shallow features H0, and expand the channel number of the feature map to twice the original.
[0043] H0=F1(Bn(I))
[0044] Where F1 represents 1x1 convolution operation, Bn represents activation function;
[0045] Step two, the extracted features are divided into X h and X l by channel splitting mechanism according to channel dimension:
[0046] X h ,X l =Split(H0)
[0047] Where Split represents channel splitting mechanism;
[0048] Step three, input X h and X l into CSPNeXt mixer feature mixer respectively to get output features Y h and Y l , CSPNeXt mixer consists of low frequency mixer and high frequency mixer, wherein high frequency mixer consists of linear layer (Linear) and depth separable convolution (DwConv), wherein low frequency mixer consists of dilated convolution (Dilated Convolution), average pooling (AvePool) and up sampling (Upsample):
[0049] Y h =DwConv(Linear(X h ))
[0050] Y1=Upsample(AvePooling(DilateConv(X l )))
[0051] Step four, concatenate the output features Y h and Y l obtained in step three according to the dimension direction of feature map to get feature Y c :
[0052] Y c =Concat(Y l ,Y h )
[0053] Where, Concat is concatenation operation, which restores the channel dimension to before splitting operation;
[0054] Step five, using 1x1 convolution to the Y of the series of high and low frequency characteristics c Again, the effective fusion of features. The feature H1 after feature fusion is input to the classification head to obtain the final classification result;
[0055] A = ClsHead(H1)
[0056] Where A represents the final classification result, and ClsHead represents the classification head.
[0057] To improve the feature representation ability of the model and balance the capture ability of the model to high and low frequency signals, we divide the feature extraction stage into four stages, and the dimension of the model channel input in each stage is twice that of the previous stage. Gradually expand the input dimension to improve the feature extraction ability; while keeping the number of channels unchanged in each stage, the channels are divided according to the different emphasis of the model in each stage of feature extraction, such as Figure 4 As shown in the figure, the model tends to extract high frequency signals (local texture) in the shallow layer, and the model tends to extract low frequency signals (global spatial information) in the deep layer, so we no longer keep the dimension consistent when designing the input to the high and low frequency signals. Instead, we input higher dimension to the high frequency mixer in the shallow layer, and input more high dimension to the low frequency mixer in the deep layer, so as to achieve the balance of high and low frequency signal extraction. The specific configuration of the model is shown in Table 4.
[0058] In this embodiment, the data set, evaluation index and implementation details are performed in sequence, as follows:
[0059] (1) Data set
[0060] We choose the public data set Imagenet-1k as the training data set, a data set widely used in computer vision, to test the performance of the network model.
[0061] (2) Evaluation index
[0062] To evaluate the effectiveness of the model on the ImageNet data set, we use accuracy, model parameter quantity (Params), and model calculation quantity GFLOPs as evaluation indexes to evaluate the image classification performance. The parameter quantity of the network model Params can be used to measure the complexity of the network model, and the more the size and number of convolution kernels, the larger the parameter quantity; while the calculation quantity of the network model GLOPs, i.e. floating point operation quantity, represents the operation speed of the network model. These three indexes will be used to evaluate the performance of the network model.
[0063] (3) Implementation details
[0064] In the experiment of the feature mixing network model, four stages are set in the feature extraction stage, and the channel number C is from 64 in the first stage to 512 in the last stage, wherein the repeated stacking number of each stage is (3, 3, 27, 3). Random cropping, horizontal flipping and vertical flipping are used to expand the training data during training. The model is trained for 300 epochs using the AdamW optimizer, the weight decay is 0.05, the learning rate lr = 1e -3 We implement the feature mixing network model on the PyTorch platform and conduct experiments using Nvidia A100 graphics cards.
[0065] (4) Analysis of experimental results
[0066] The purpose of this experiment is to verify the superior performance of the feature mixing network model in the image classification task on the ImageNet dataset, and to verify the superiority of the model. The heat map visualization of the results of the model and the comparison with other models are as follows Figure 3
[0067] Other CNN or Transformer-based models are tested and compared on the ImageNet dataset, and the precision, parameter quantity Params and calculation quantity GFLOPs of each model on the dataset are given in Table 1. The heat map visualization results of the Figure 3 In the heat map visualization results, we can see that our model can effectively consider global and local information, for example, in the wombat image, our model can skip the arm and accurately focus on the entire wombat, including the four limbs.
[0068] Table 1 shows the performance of different types of models on the ImageNet-1K classification. All these models are trained only on the ImageNet-1K training set, and the accuracy of the validation set is reported
[0069]
[0070]
[0071] This embodiment verifies the effectiveness of the kernel size and channel splitting mechanism in the proposed CSPNeXt mixer and high-frequency mixer through ablation experiments.
[0072] First, we analyze the impact of the CSPNeXt mixer model on network performance, and then remove each module in the CSPNeXt mixer accordingly, and then report the corresponding results in Table 2. The checkmark represents whether the corresponding branch is used. It is worth noting that we achieve excellent performance using DwConv alone combined with average pooling instead of using convolution alone. This shows that our CSPNeXt mixer is indeed effective.
[0073] Table 2. Ablation study of CSPNeXt mixer on ImageNet-1K. All models are trained for 200 epochs.
[0074]
[0075] To verify the benefits of large kernel convolution, we tried several kernel sizes, including 3, 5, 7, and 9. The performance of the network improved from 79.4% (3x3) to 80.6% (7x7), while the FLOPs of the network remained unchanged, as shown in Table 3. In addition, the gain of larger kernels saturates at 7x7.
[0076] Table 3. Ablation study of different kernel sizes on CSPNeXt mixer on ImageNet-1K. All models are trained for 200 epochs.
[0077]
[0078] To verify the effectiveness of the new channel splitting mechanism, we designed various channel division schemes, and the detailed configurations are shown in Table 4. From the table, we can see that CSPNeXt-S3 uses more high-frequency components at the shallow layer and more deep-layer low-frequency component dimensions, and the overall performance of the model is better than the original CSPNeXt-S1.
[0079] Table 4. Ablation study of different channel divisions on CSPNeXt mixer on ImageNet-1K. Among them, CSPNeXt-S1 is the original CSPNeXt mixer, and CSPNeXt-S2 is the CSPNeXt mixer with the new channel splitting mechanism. h represents the dimension of the input high-frequency component, C l represents the dimension of the input low-frequency component, and C represents the dimension of the unsegmented feature map. All models are trained for 200 epochs.
[0080]
[0081]
[0082] The application proposes a feature mixing network model, feature extraction is composed of four stages, each stage is composed of a channel splitting mechanism and a high-low frequency mixer, and the high-low frequency mixer is used for processing high-low frequency signals of features. In addition, the channel splitting mechanism is designed to balance the ability of the model to process high-low frequency signals at different stages. A large number of experiments on the ImageNet dataset show that the application is superior to other advanced network models in accuracy and computational complexity.
[0083] The specific embodiments described herein merely illustrate the spirit of the application. Those skilled in the art of the application can make various modifications or supplements to the described specific embodiments or replace them with similar ways, without departing from the spirit of the application or exceeding the scope defined by the appended claims.
Claims
1. A method for image classification using a feature fusion network model, characterized in that: Includes the following steps: Step 1: Given an image I as input, use a 1×1 convolutional layer and activation function to extract shallow features H0, and double the number of channels in the feature map: H0=F1(Bn(I)) Where F1 represents a 1×1 convolution operation and Bn represents the activation function; Step 2: Divide the extracted feature map into X channels according to the channel dimension using a channel splitting mechanism. h and X l : X h ,X l =Split(H0) Split represents the channel splitting mechanism; Step 3, X h and X l The inputs are fed into multiple CSPNEXt mixers, and each CSPNEXt mixer produces an output feature Y. h and Y l Each CSPNEXt mixer feature mixer consists of a low-frequency mixer and a high-frequency mixer, where the high-frequency mixer includes linear layers and depthwise separable convolutional layers, and the low-frequency mixer includes dilated convolution, average pooling, and upsampling. Y h =DwConv(Linear(X h )) Y1=Upsample(AvePooling(DilateConv(X l ))) Step 4: The output feature Y obtained in Step 3 is processed through a concatenation operation. h and Y l Feature Y is obtained by concatenating the feature maps along their dimensional directions. c : AND c =Concat(Y l ,AND h ) Concat is a concatenation operation that restores the channel dimension to its state before the splitting operation. Step 5: Apply 1×1 convolution to the Y-axis of the cascaded high and low frequency features. c The features are then effectively fused again, and the fused feature H1 is input into the classification head to obtain the final classification result. A = ClsHead(H1) Where A represents the final classification result, and ClsHead represents the classification head.
2. The method for image classification using a feature fusion network model as described in claim 1, characterized in that: Each CSPNEXt mixer features a mixer that includes a high-frequency mixer and a low-frequency mixer.
3. The method for image classification using a feature fusion network model as described in claim 2, characterized in that: The high-frequency mixer contains multiple DwConv convolutional and linear layers.
4. The method for image classification using a feature fusion network model as described in claim 2, characterized in that: The low-frequency mixer comprises multiple DilatedConv convolutions, average pooling layers, and upsampling layers.
5. The method for image classification using a feature fusion network model as described in claim 3, characterized in that: The channel splitting mechanism works as follows: more channel dimensions are allocated to the high-frequency mixer in shallow layers of the model, and more channels are allocated to the low-frequency mixer in deep layers of the model. Different layers exhibit different capture capabilities for high and low frequency signals.
6. The method for image classification using a feature fusion network model as described in claim 2, characterized in that: The high-frequency mixer first processes the input feature X... h Linear projection is performed using a Liner, followed by a DwConv with a kernel size of 7×7.
7. The method for image classification using a feature fusion network model as described in claim 5, characterized in that: Each low-frequency mixer first processes the input feature X l The global receptive field is obtained through Dilated Convolution, and then global average pooling is used for global feature interaction while reducing the width and height of the feature map. Finally, the scale of the feature map is restored through an upsampling layer.
8. The method for image classification using a feature fusion network model as described in claim 1, characterized in that: The feature fusion network model uses the Adam optimization algorithm during training, with a minimum batch size of 400 and an initial learning rate of 10. e-3 In 200 iterations, the initial learning rate was reduced to one-tenth of its original value, for a total of 300 iterations.
9. The method for image classification using a feature fusion network model as described in claim 1, characterized in that: Data augmentation is performed on the input image I using random cropping, horizontal flipping, and vertical flipping to expand the training data.
10. The method for image classification using a feature fusion network model as described in claim 1, characterized in that: The feature hybrid network model needs to be implemented as a multi-channel aggregation network on the PyTorch platform.
11. The method for image classification using a feature fusion network model as described in claim 1, characterized in that: The activation function is either the LeakyReLU function or the ReLU function.