General image classification method based on BlendNeXt model

Abstract

The invention discloses a general image classification method based on a BlendNeXt model. The method comprises the following steps: S1, obtaining a public fruit image; s2, preprocessing the image, and constructing a data set; s3, a BlendNeXt model is constructed; s4, the BlendNeXt is trained, and the weight of the model is saved; and S5, evaluating the performance of the model on the test set. The BlendNeXt is a brand-new computer vision backbone model, the dilemma that a traditional backbone model cannot still keep high accuracy under the condition of low model parameter quantity is relieved, due to the fact that the model parameter quantity is low, the training cost is low, the BlendNeXt can be applied to various picture classification scenes, the classification error rate is reduced, and the efficiency is improved.

Description

Technical Field

[0001] The design of the present invention belongs to the field of computer vision and deep learning technology, specifically a general image classification method based on the BlendNeXt model. Background Art

[0002] In deep learning models for computer vision, the model architecture typically consists of a feature extraction backbone network (Backbone) and a task-oriented head (Head). As the core component of visual representation learning, the Backbone's structural design directly affects the performance ceiling of downstream tasks such as image classification, object detection, and semantic segmentation. Currently, the mainstream Backbone architecture presents three major technical approaches:

[0003] 1. Convolutional Neural Network (CNN) Architecture

[0004] Convolutional backbones, such as ResNet and EfficientNet, construct feature pyramids through hierarchical local receptive fields. Their advantages lie in translational equivariance and hardware friendliness, making them particularly suitable for edge device deployment. However, they are limited by local inductive bias and have difficulty modeling long-range dependencies within a limited parameter scale, resulting in a significant diminishing return on model performance and parameter size.

[0005] 2. Pure Transformer Architecture

[0006] Visual Transformers such as the Vision Transformer (ViT) achieve dynamic receptive fields through a global self-attention mechanism, demonstrating modeling capabilities that surpass CNNs on large datasets such as ImageNet. However, their computational complexity increases quadratically with sequence length and is sensitive to the size of training data. They are prone to attention drift in small sample sizes. Research has shown that ViT requires significantly more training data and time than CNNs to achieve comparable performance.

[0007] 3. Hybrid Backbone

[0008] Emerging Conv-Transformer hybrid architectures, such as CoAtNet, employ a staged feature fusion strategy: convolution at the bottom layer extracts high-resolution local features, while multi-head attention at the higher layers builds global semantic associations. This design combines the spatial inductive bias of CNNs with the global modeling capabilities of Transformers. On the ImageNet-1K benchmark, CoAtNet achieved a top-1 accuracy of 83.64% while maintaining real-time inference efficiency on mobile devices.

[0009] In order to solve the problem that the existing convolutional model and self-attention model cannot achieve a balance between the number of parameters, model performance and training cost, it is necessary to propose a new general image classification model. Summary of the Invention

[0010] The general image classification method based on the BlendNeXt model proposed in this invention aims to solve the problem that existing convolutional models and attention models cannot achieve a balance between parameter quantity, model performance and training cost. The general image classification method based on the BlendNeXt model adopts the following technical solution: It is characterized by comprising the following steps:

[0011] S1: Obtain public fruit images;

[0012] S2: Preprocess the fruit images to construct a dataset and divide it into training set, validation set and test set;

[0013] S3: Build the BlendNeXt model;

[0014] The BlendNeXt model uses MetaNeXt as its basic framework. The framework mainly consists of an image embedding layer, a feature encoding network, and a category prediction network. The feature encoding network of this framework consists of four stages. Each stage has a multi-scale deep convolutional module (MetaNeXt Block) composed of several token-mixers and MLPs. A cross-stage fusion module is added to address the spatial information loss caused by link downsampling.

[0015] S4: Train BlendNeXt and save model weights;

[0016] Use the training data of S2 to train the BlendNeXt model, and use the preprocessed validation set to monitor the model training process, adjust the hyperparameters and save the model weights;

[0017] S5: Evaluate model performance on the test set;

[0018] Evaluate model formation on the test set: The model loads the model weights saved in the previous step and uses the test set to test the model performance.

[0019] Furthermore, the construction of the BlendNeXt model in the above scheme includes:

[0020] Build the image embedding layer: It consists of a 3*3 2D convolution module, BatchNorm function and GELU function, which is used to convert the original image into a high-dimensional feature map;

[0021] Construct an adaptive multi-scale deep convolution module (BlendNeXt Block): The high-dimensional feature map is expanded in number of channels through a fully connected layer, and then the Splite function is used to split the feature map into four parts. The first part is used for identity mapping, the second part is used for square convolution, and the third and fourth parts are used for orthogonal strip convolution. The size of the square convolution kernel is specified by a hyperparameter, and the size of the strip convolution kernel is calculated by the model based on the square convolution kernel size and does not need to be manually specified.

[0022] The four feature maps after convolution are spliced through channels to obtain a new feature map. The new feature map is activated after channel feature optimization, and finally the original number of channels is restored through a fully connected layer. The module process formula 1:

[0023]

[0024] The process of channel feature optimization is as follows: perform average pooling and maximum pooling on the feature map x, then concatenate the results, and then convolve the concatenated two-channel feature map to obtain a one-channel feature map t. Finally, add the feature map x and the scaled value of the difference between the feature map x and the feature map t to obtain the output. The model flow chart is Formula 2:

[0025] t=Conv(Concat(AvgPool(x),MaxPool(x))

[0026] ChannelFeatureRefine(x)=x+β·(xt) (2)

[0027] Where x is the input feature map, OUTPUT is the output of BlendNeXt Block, Conv is the depthwise convolution module, its subscripts represent the width and height of the convolution kernel, Splitte is the channel splitting operation, Concat is the channel concatenation operation, F1 and F2 are linear functions, GELU is a nonlinear activation function, MaxPool is the maximum pooling operation, AvgPool is the average pooling operation, and β is a learnable parameter;

[0028] Construct a cross-stage feature fusion module: It consists of a lightweight cross-layer fusion module SSFF-L (Skip Stage Feature Fusion Light) and a heavyweight cross-stage feature fusion module SSFF-H (Skip Stage Feature Fusion Heavy), where;

[0029] The lightweight cross-layer fusion module SSFF-L adjusts the number of channels of the output features of the previous stage to the number of channels of the input features of the next stage through point convolution, aligns the width and height of the two-stage features through adaptive pooling, and finally linearly fuses the two-stage features. The process is shown in Equation 3:

[0030] SSFF Light (X N , X N+1 )=Avg(Conv 1×1 (X N ))+X N+1 (3)

[0031] The heavyweight cross-layer fusion module is implemented through cross-multi-head self-attention. The input feature map of the latter stage is q, and the feature map of the previous stage is k, v. Position encoding is added. While fusing features, the connection between local and global is established to achieve global modeling. The process is shown in Equation 4:

[0032] Q=W q (X N+1 )+PosEmbed N+1

[0033] K=W k (X N )+PosEmbed N

[0034] V=W v (X N )

[0035]

[0036] Among them, X N is the feature of the Nth layer, X N+1 It is the feature of N+1 layer, Conv is the convolution operation, Avg is the adaptive pooling operation, Wq, Wk and Wv are mapping matrices, Proj is the fully connected layer, and PosEmbed is the position encoding;

[0037] Constructing a category prediction network: The category prediction network contains a global pooling function, a fully connected layer, and a softmax function. The global pooling function converts the feature map into a one-dimensional vector, which is then input into the fully connected layer and the softmax function to obtain the probability that the sample belongs to each category.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) BlendNeXt moves the first fully connected layer in the multi-layer perceptron to before the channel splitting operation, first expanding the number of channels in the feature map and then splitting it. In this way, the number of channels in the feature map obtained by each branch increases, which is stronger than the feature representation capability of the existing model;

[0040] (2) The BlendNeXt model is divided into four stages. The early stage uses a large square convolution kernel, and the later stage uses a small square convolution kernel. The strip convolution kernel is automatically calculated by the size of the square convolution kernel. Compared with the fixed square convolution kernel and fixed strip convolution kernel of the existing model, the adaptability to the smaller feature maps is improved.

[0041] (3) The channel feature optimization proposed by BlendNeXt can reorganize the features of the channel dimension and optimize the feature representation. Compared with the existing channel feature optimization module, it abandons the practice of using convolution to directly compress n-channel features into 1-channel features. Instead, it first calculates the maximum and average values, and then superimposes the maximum and average values, and uses convolution to compress 2 channels into 1-channel features, thereby reducing the number of parameters while ensuring performance.

[0042] (4) The cross-channel feature fusion module (SSFF) proposed by BlendNeXt makes full use of the input features of each stage. Compared with the existing model that only relies on the output of the last stage for classification, it not only improves the performance of the model but also increases the utilization of feature maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flowchart of the present invention

[0044] Figure 2 Schematic diagram of BlendNeXt model working

[0045] Figure 3 Schematic diagram of the adaptive multi-scale deep convolution module (BlendNeXt Block)

[0046] Figure 4 Schematic diagram of the cross-stage feature fusion module (SSFF Block)

[0047] Figure 5 This is the classification effect diagram of CIFAR100 in Example 1

[0048] Figure 6 This is the classification effect diagram of STL10 in Example 2

[0049] Figure 7 Schematic diagram of the classification of fruits 100 in Example 3 DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings.

[0051] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they should be protected by patent law.

[0052] Example 1:

[0053] Reference Figure 1 , the general image classification method based on the BlendNeXt model includes the following steps:

[0054] S1: Select the public dataset CIFAR100 on Kaggle;

[0055] S2: Download the CIFAR100 dataset, preprocess the images, serialize the image files in the CIFAR100 dataset into .pkl files, and then divide the data into training, validation, and test sets.

[0056] S3: Building BlendNeXt:

[0057] 3.1 Constructing the image embedding layer: The image embedding layer consists of two groups of 3*3 2D convolution modules, BatchNorm function and GELU function in sequence;

[0058] 3.2 Constructing the Adaptive Multi-Scale Deep Convolution Module (BlendNeXt Block): The BlendNeXt Block consists of the first fully connected layer, the Split function, a four-branch structure (the first branch is the identity mapping, the second branch is the square convolution, and the third and fourth branches are mutually orthogonal banded convolutions), the concat function, the activation function, and the second fully connected layer.

[0059] 3.3 Constructing cross-stage feature fusion module: The lightweight cross-layer feature fusion module (SSFF-L) is composed of adaptive pooling, point convolution and addition in sequence; the heavyweight cross-layer feature fusion module (SSFF-H) is composed of point convolution, matrix multiplication, addition, softmax function, and position encoding. For details, refer to Figure 4 SSFF-H part;

[0060] 3.4 Constructing a category prediction network: The category prediction network is composed of a global pooling function, a fully connected layer, and a softmax function in sequence;

[0061] S4: Train BlendNeXt and save the model weights, use the training set to train the model, use the validation set to monitor the training process, adjust the hyperparameters and save the model weights;

[0062] The training environment uses an NVIDIA 4090 graphics card and an Intel Xeon Gold 5318Y processor. The training software environment is Ubuntu 20.04, CUDA Version 12.4, PyTorch 1.13, and Python 3.10. Training settings: batch size is set to 256, the maximum learning rate is 1e-3, and the minimum setting is 1e-6. The cosine scheduler is used to automatically adjust the learning rate, and the number of training epochs is set to 100.

[0063] Table 1: Top-1 and Top-3 accuracy of 6 models on the CIFAR100 dataset

[0064]

[0065] S5: This example uses the preprocessed test set to test the model obtained in S4. The classification results are shown in Table 1. The model classification visualization effect is shown in Figure 5 shown.

[0066] Example 2:

[0067] Reference Figure 1 , the general image classification method based on the BlendNeXt model includes the following steps:

[0068] S1: Select the public dataset STL10 on Kaggle;

[0069] S2: Download the STL10 dataset, preprocess the images, serialize the image files in the STL10 dataset into pkl files, and then divide the data into training set, validation set, and test set;

[0070] S3: Building BlendNeXt:

[0071] S3.1: Constructing the image embedding layer: The image embedding layer consists of two sets of 3*3 2D convolution modules, BatchNorm function and GELU function in sequence;

[0072] S3.2: Constructing the Adaptive Multi-Scale Deep Convolutional Module (BlendNeXt Block): The BlendNeXt Block consists of the first fully connected layer, the Split function, a four-branch structure (the first branch is the identity mapping, the second branch is the square convolution, and the third and fourth branches are mutually orthogonal banded convolutions), the concat function, the activation function, and the second fully connected layer.

[0073] S3.3: Construct cross-stage feature fusion module: The lightweight cross-layer feature fusion module (SSFF-L) is composed of adaptive pooling, point convolution and addition in sequence; the heavyweight cross-layer feature fusion module (SSFF-H) is composed of point convolution, matrix multiplication, addition, softmax function, and position encoding. For details, refer to Figure 4 SSFF-H part;

[0074] S3.4: Construct a category prediction network: The category prediction network is composed of a global pooling function, a fully connected layer, and a softmax function in sequence;

[0075] S4: Train BlendNeXt and save the model weights. The model is trained using the training set, and the validation set is used to monitor the training progress. Hyperparameters are adjusted and the model weights are saved. The training environment uses an NVIDIA 4090 graphics card and an Intel Xeon Gold 5318Y processor. The training software environment is Ubuntu 20.04, CUDA Version 12.4, PyTorch 1.13, and Python 3.10. Training settings: batch size is set to 256, the maximum learning rate is 1e-3, and the minimum setting is 1e-6. The cosine scheduler is used to automatically adjust the learning rate, and the number of training epochs is set to 100.

[0076] Table 2: Top-1 and Top3 accuracy of 6 models on the STL10 dataset

[0077]

[0078] S5: This example uses the preprocessed test set to test the model obtained in S4. The classification results are shown in Table 2. The model classification visualization effect is shown in Figure 6 shown.

[0079] Example 3:

[0080] Reference Figure 1 , the general image classification method based on the BlendNeXt model includes the following steps:

[0081] S1: Select the public dataset fruits100 on Kaggle;

[0082] S2: Download the fruits100 dataset, preprocess the images, convert the images of different sizes in the fruits100 dataset into jpg files with a resolution of 64*64, serialize the jpg files into pkl files, and then divide the data into training set, validation set and test set;

[0083] S3: Building BlendNeXt:

[0084] S3.1: Constructing the image embedding layer: The image embedding layer consists of two sets of 3*3 2D convolution modules, BatchNorm function and GELU function in sequence;

[0085] S3.2: Constructing the Adaptive Multi-Scale Deep Convolutional Module (BlendNeXt Block): The BlendNeXt Block consists of the first fully connected layer, the Split function, a four-branch structure (the first branch is the identity mapping, the second branch is the square convolution, and the third and fourth branches are mutually orthogonal banded convolutions), the concat function, the activation function, and the second fully connected layer.

[0086] S3.3: Construct cross-stage feature fusion module: The lightweight cross-layer feature fusion module (SSFF-L) is composed of adaptive pooling, point convolution and addition in sequence; the heavyweight cross-layer feature fusion module (SSFF-H) is composed of point convolution, matrix multiplication, addition, softmax function, and position encoding. For details, refer to Figure 4 SSFF-H part;

[0087] S3.4: Construct a category prediction network: The category prediction network is composed of a global pooling function, a fully connected layer, and a softmax function in sequence;

[0088] S4: Train BlendNeXt and save the model weights. Use the training set to train the model, use the validation set to monitor the training process, adjust hyperparameters, and save the model weights. This example uses an NVIDIA 4090 graphics card and an Intel Xeon Gold 5318Y processor. The training software environment is Ubuntu 20.04, CUDA Version: 12.4, PyTorch 1.13, and Python 3.10. Training settings: batch size is set to 256, the maximum learning rate is 1e-3, the minimum setting is 1e-6, the cosine scheduler is used to automatically adjust the learning rate, and the number of training epochs is set to 100.

[0089] Table 3: Top-1 and Top3 accuracy of 6 models on the fruits100 dataset

[0090]

[0091]

[0092] S5: This example uses the preprocessed test set to test the model obtained in S4. The classification results are shown in Table 3. The model classification visualization effect is shown in Figure 7 shown.

Claims

1. A general image classification method based on the BlendNeXt model, characterized by: The following steps are involved: S1: Obtain public fruit images; S2: Preprocess the fruit images to construct a dataset and divide it into training set, validation set and test set; S3: Build the BlendNeXt model; The BlendNeXt model uses MetaNeXt as its basic framework. The framework mainly consists of an image embedding layer, a feature encoding network, and a category prediction network. The feature encoding network of this framework consists of four stages. Each stage has a multi-scale deep convolutional module (MetaNeXt Block) composed of several token-mixers and MLPs. A cross-stage fusion module is added to compensate for the loss of spatial information caused by downsampling. S4: Train BlendNeXt and save model weights; Use the training data of S2 to train the BlendNeXt model, and use the preprocessed validation set to monitor the model training process, adjust the hyperparameters and save the model weights; S5: Evaluate model performance on the test set; Evaluate model formation on the test set: The model loads the model weights saved in the previous step and uses the test set to test the model performance.

2. The general image classification method based on the BlendNeXt model according to claim 1, characterized in that: The construction of the BlendNeXt model includes: Build the image embedding layer: It consists of a 3*3 2D convolution module, BatchNorm function and GELU function, which is used to convert the original image into a high-dimensional feature map; Construct an adaptive multi-scale deep convolution module (BlendNeXt Block): The high-dimensional feature map is expanded in number of channels through a fully connected layer, and then the Splite function is used to split the feature map into four parts. The first part is used for identity mapping, the second part is used for square convolution, and the third and fourth parts are used for orthogonal strip convolution. The size of the square convolution kernel is specified by a hyperparameter, and the size of the strip convolution kernel is calculated by the model based on the square convolution kernel size and does not need to be manually specified. The four feature maps after convolution are spliced through channels to obtain a new feature map. The new feature map is activated after channel feature optimization, and finally the original number of channels is restored through a fully connected layer. The module process formula 1: The process of channel feature optimization is as follows: perform average pooling and maximum pooling on the feature map x, then concatenate the results, and then convolve the concatenated two-channel feature map to obtain a one-channel feature map t. Finally, add the feature map x and the scaled value of the difference between the feature map x and the feature map t to obtain the output. The model flow chart is Formula 2: t=Conv(Concat(AvgPool(x),MaxPool(x)) ChannelFeatureRefine(x)=x+β·(xt) (2) Where x is the input feature map, Output is the output of BlendNeXt Block, Conv is the depthwise convolution module, its subscripts represent the width and height of the convolution kernel, Split is the channel splitting operation, Concat is the channel concatenation operation, F1 and F2 are linear functions, GELU is a nonlinear activation function, MaxPool is the maximum pooling operation, AvgPool is the average pooling operation, and β is a learnable parameter; Construct a cross-stage feature fusion module: It consists of a lightweight cross-layer fusion module SSFF-L (Skip Stage Feature Fusion Light) and a heavyweight cross-stage feature fusion module SSFF-H (Skip Stage Feature Fusion Heavy), where; The lightweight cross-layer fusion module SSFF-L adjusts the number of channels of the output features of the previous stage to the number of channels of the input features of the next stage through point convolution, aligns the width and height of the two-stage features through adaptive pooling, and finally linearly fuses the two-stage features. The process is shown in Equation 3: SSFF Light (X N ,X N+1 )=Avg(Conv 1×1 (X N ))+X N+1 (3) The heavyweight cross-layer fusion module is implemented through cross-multi-head self-attention. The input feature map of the latter stage is q, and the feature map of the previous stage is k, v. Position encoding is added. While fusing features, the connection between local and global is established to achieve global modeling. The process is shown in Equation 4: Q=W q (X N+1 )+PosEmbed N+1 K=W k (X N )+PosEmbed N V=W v (X N ) Among them, X N is the feature of the Nth layer, X N+1 It is the feature of N+1 layer, Conv is the convolution operation, Avg is the adaptive pooling operation, Wq, Wk and Wv are mapping matrices, Proj is the fully connected layer, and PosEmbed is the position encoding; Constructing a category prediction network: The category prediction network contains a global pooling function, a fully connected layer, and a softmax function. The global pooling function converts the feature map into a one-dimensional vector, which is then input into the fully connected layer and the softmax function to obtain the probability that the sample belongs to each category.

Images