A spatial-context-based multi-branch image classification method

Abstract

This invention belongs to the field of image processing technology, specifically a spatial-context-based multi-branch image classification method. The invention includes data preprocessing, dataset partitioning, model training, model testing, and model performance verification. This invention employs a synergistic effect of spatial paths with rich spatial information and contextual paths with rapid downsampling, achieving a proper balance between speed and classification performance, while simultaneously realizing high accuracy and real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multi-branch image classification method based on spatial-context. Background Technology

[0002] Image classification is a crucial task in computer vision, aiming to train a model that can correctly categorize images into predefined classes based on their features. Its applications are widespread, including scene recognition, face recognition, object recognition, and video surveillance. It plays a vital role in security, autonomous driving, and other fields, and with continuous technological advancements, image classification will be applied to even more areas.

[0003] Deep learning-based methods have made significant progress in image classification tasks, especially the application of convolutional neural networks (CNNs). CNNs can automatically learn features from images without requiring manual feature design, significantly improving classification accuracy. LeCun et al. proposed the first truly production-ready neural network model, LeNet-5. AlexNet won the ImageNet image classification competition that year with a decisive advantage, and since then, neural networks have received widespread attention from researchers in computer vision tasks.

[0004] Attention mechanisms are designed to mimic the human visual attention process and are widely used in computer vision tasks. They assign different weights to information at different locations in an image, allowing the model to focus more on parts relevant to the current task. Its advantages include increased attention to important regions of the image, reduced interference from redundant information, and improved robustness and generalization ability.

[0005] In fields such as product recognition, object recognition in autonomous driving, automated inspection, and quality control, there is a high demand for classification accuracy and efficient reasoning speed. Currently, several classification methods exist:

[0006] (1) The computational complexity is reduced by cropping or resizing the image, but most of the spatial information of the original image is lost;

[0007] (2) Improving inference speed by pruning network channels also compromises spatial information;

[0008] (3) Abandon the final downsampling operation of the network, but the receptive field of the model is not enough to cover large targets, resulting in poor recognition ability;

[0009] (4) By using dilated convolution and large kernels to capture enough receptive field encoding space information, but these operations have computational requirements and memory consumption, resulting in low speed.

[0010] In summary, image classification is an important task with a wide range of applications. However, existing deep learning-based image classification methods struggle to simultaneously meet the requirements of real-time performance and high accuracy, which is not conducive to practical application. Summary of the Invention

[0011] To overcome the shortcomings of the existing technology, this invention provides a spatial-context-based multi-branch image classification method. This invention employs a spatial path rich in spatial information and a contextual path with rapid downsampling to achieve a proper balance between speed and classification performance, while simultaneously realizing high accuracy and real-time performance.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A spatial-context-based multi-branch image classification method specifically includes the following steps:

[0014] S1. Data preprocessing: Performing preprocessing operations on the input data;

[0015] S2. Dataset partitioning: Divide the preprocessed data into training set, validation set and test set according to the proportions;

[0016] S3. Model Training: Train the entire multi-branch image classification model using the pre-defined training set until the loss converges; Given an input image X, extract features through spatial path and context path respectively and perform feature fusion.

[0017] S4. Model Testing: Test the classification model using the predefined test set and training weights, and output the results;

[0018] S5. Model Performance Verification: Calculate performance evaluation metrics for the output results. If the performance is good, training ends; otherwise, modify the model parameters and retrain.

[0019] Preferably, in step S3, the ResNet34 model is first used to extract the depth features of the input image in the context path, and then the global context module is used to obtain high-level context features based on the depth features.

[0020] The preferred formula for depth features is:

[0021] ;

[0022] X∈R 224x224x3 ;

[0023] in, This represents the number of channels in the i-th feature extraction layer of the ResNet34 model. The features extracted by ResNet34 are represented by H, W, and C, which represent the height, width, and number of channels of the image, respectively. R indicates that it is an RGB image, 224 represents the image size, and 3 represents the number of channels.

[0024] Preferably, the formula for the global context module is:

[0025]

[0026]

[0027] in, Indicates the global context module; and All are input features; Represents a 1x1 convolution; ReLU represents an activation function; LN represents layer normalization. For contextual features; These are the features of the last layer of ResNet34.

[0028] Convolutional and pooling layers are used to extract the deep features required for classification.

[0029] Preferably, the spatial path shares the first three layers of the ResNet34 model, and the extracted output feature map is 1 / 8 of the original image; the formula for the output feature extracted by the spatial path is:

[0030]

[0031] in, Features extracted for spatial paths; is the third layer feature of ResNet34; X represents the input image.

[0032] Preferably, the spatial path includes a spatial attention module that enables the network to extract key features with spatial location recognition; the contextual path includes a channel attention module that captures channel-level dependencies and important feature correlations between different feature channels.

[0033] Preferably, in the spatial attention module, the input features are given first. Two features are obtained using global average pooling and global max pooling. , Spatial attention maps are then generated using a shared multilayer perceptron and sigmoid activation function. Finally, multiply with the original image; where H and W represent the image height and width respectively, and C represents the number of channels. Represents spatial input features, This represents the spatial global average pooling feature. Represents the global max pooling feature in space. Represents a spatial attention map;

[0034] The formula The formula The formulas are as follows:

[0035]

[0036]

[0037]

[0038]

[0039] In the formula, Key features of spatial attention; This represents the sigmoid operation; MLP stands for Multilayer Perceptron. Indicates a channel splicing operation; This indicates a global average pooling operation; This indicates a global max pooling operation.

[0040] Preferably, in the channel attention module, the input features are given first. Features are obtained through global average pooling. Subsequently, a channel attention map is generated using a multilayer perceptron and a sigmoid activation function. Finally, multiply by the original image; where, Indicates channel input characteristics, This represents the global average pooling characteristic of the channel. This represents the channel attention map, where H and W represent the height and width of the image, respectively, and C represents the number of channels;

[0041] The formula The formulas are as follows:

[0042]

[0043]

[0044]

[0045]

[0046] in, Key features representing channel attention; H represents the sigmoid operation; MLP stands for Multilayer Perceptron; H represents the height of the image; W represents the width of the image. i, j Represents pixels; This is a characteristic of initial fusion; This indicates element-wise addition.

[0047] Preferably, in step S3, the features after feature fusion undergo depthwise convolution and pointwise convolution; the feature formulas for depthwise convolution and pointwise convolution are as follows:

[0048]

[0049]

[0050]

[0051] in, i, j Both represent the spatial location of the output feature map; Features representing depthwise convolution; s Represents the channel index; m, n Indicates the spatial location of the convolution kernel; The weights represent the convolution kernel; Features of pointwise convolution; Represents the channel index before pointwise convolution; Represents the channel index after pointwise convolution; The weights represent the convolution kernel; This is a characteristic of the final fusion.

[0052] Preferably, in step S1, the preprocessing operation includes one or more of the following: random rotation of the input data, size transformation, random vertical and horizontal flipping, and color gamut distortion.

[0053] Preferably, in step S2, the preprocessed data is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0054] The advantages of this invention are:

[0055] (1) The present invention adopts the synergistic effect of spatial path with rich spatial information and context path with fast downsampling to achieve a correct balance between speed and classification performance, while achieving high accuracy and real-time performance; in the fields of product recognition, object recognition in autonomous driving, automated detection and quality control, it can achieve a balance between speed and classification performance, and meet the requirements of real-time performance and high accuracy.

[0056] (2) The present invention designs different attention mechanisms for two paths to obtain discriminative features in order to improve the classification performance of different categories of images; among them, the spatial attention module is used to enable the network to extract key features with spatial location recognition, so as to consider the importance of different locations in the extracted spatial information; and the channel attention module is used to capture channel-level dependencies and important feature correlations between different feature channels, which can suppress interference from irrelevant regions in the context features.

[0057] (3) In order to enable the network to better focus on the prominent target region, this invention fuses the attention features of spatial path and context path. The fused features not only make full use of spatial feature information, but also channel feature information, which can obtain category-related spatial context features and suppress redundant features unrelated to the target. Through the fusion of the two-branch network, the extracted features are more robust and discriminative than those of the single-path network.

[0058] (4) In order to improve accuracy without sacrificing speed, we also studied parallel depthwise convolution (DW) and pointwise convolution (PW). After the fused features are processed by DW and PW, the feature information of each channel is extracted in detail, while the channel relationship between features is preserved. This can significantly reduce the number of convolution kernels and the amount of computation while ensuring classification accuracy.

[0059] (5) This invention can be used for image classification tasks in various industrial fields, such as face recognition and video surveillance. The acquired image data extracts rich features through complementary spatial and contextual paths, which is fast and accurate. The design of spatial attention modules and channel attention modules can help enhance the target region, and the features obtained by dual-branch attention fusion are more robust. Replacing traditional convolution with depthwise convolution and pointwise convolution reduces the number of parameters and computation while ensuring classification accuracy. Attached Figure Description

[0060] Figure 1 This is a structural diagram of the model of the present invention.

[0061] Figure 2 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] like Figure 1-2 As shown, a spatial-context-based multi-branch image classification method includes the following steps:

[0064] S1. Data preprocessing: Preprocessing operations include random rotation, size transformation, random vertical and horizontal flipping, and color gamut distortion of the input data;

[0065] S2. Dataset partitioning: Divide the preprocessed data into training set, validation set and test set in a ratio of 8:1:1;

[0066] S3. Model Training: Train the entire multi-branch image classification model using the pre-defined training set until the loss converges; given the input image X∈R 224x224x3 It extracts features and fuses features through spatial path and context path, and outputs the category and probability; where X represents the input image, R represents that it belongs to RGB image, 224 represents the image size, and 3 represents the number of channels;

[0067] S4. Model Testing: Test the classification model using the predefined test set and training weights, and output the results;

[0068] S5. Model Performance Verification: Calculate performance evaluation metrics for the output results. If the performance is good, training ends; otherwise, modify the model parameters and retrain.

[0069] Specifically, the overall classification model has the following structure: Figure 1 As shown. Given an input image X∈R 224x224x3 (224 represents the image size, 3 represents the number of channels), through the spatial path ( ) and context path ( Features are extracted and fused, and the category and probability are output. The specific implementation methods for feature extraction and feature fusion are as follows:

[0070] (1) Feature extraction.

[0071] Context path ( Drawing on existing research findings, we use a pre-trained ResNet34 model with good classification performance as a fast downsampling classification model to extract rich deep features. We discard all fully connected layers and softmax, and only use convolutional layers and pooling layers to extract the features required for classification.

[0072]

[0073] in, The features extracted by ResNet34 are H, W, and C, which represent the height, width, and number of channels of the image, respectively, and X represents the input image.

[0074] Then, a global context module with very few parameters and low computational cost is added to the end of the model. Using the last feature extraction layer of ResNet34 as input, high-level contextual features are obtained. It is important to note that, in order to keep the model lightweight, we did not use multi-scale feature extractors such as ASPP with a large number of parameters.

[0075]

[0076]

[0077] in, The global context module contains global attention pooling and bottleneck transformations that capture channel-level dependencies. and For input features, For 1x1 convolutions, ReLU represents the activation function, and LN represents the layer normalization operation. For contextual features, This is the last layer of features in ResNet34, followed by a... The module, its input is ,Right now, It is both the output feature of ResNet34 and Input characteristics of the module.

[0078] To compensate for the shallow feature information, we established another branch of the spatial path ( It shares the first three layers of parameters of the ResNet34 model, and the extracted output feature maps are 1 / 8 of the original image, which preserves rich spatial details and reduces the parameters of the entire network.

[0079]

[0080] in, Features extracted for spatial paths These are features from the third layer of ResNet34.

[0081] We designed different attention mechanisms (including spatial attention and channel attention) in two paths to obtain discriminative features, thereby improving the classification performance of different image categories.

[0082] (1) To consider the importance of different locations in the extracted spatial information, we utilize a spatial attention module to enable the network to extract key features with spatial location recognition capabilities. Given input features (H and W represent the height and width of the image, respectively, and C is the number of channels.) We use global average pooling and global max pooling to obtain two features. , Spatial attention maps are then generated using a shared multilayer perceptron and a sigmoid activation function. Finally, multiplying it with the original image highlights the ability to represent spatial features.

[0083]

[0084]

[0085]

[0086]

[0087] in, Key features of spatial attention This represents the sigmoid operation; MLP stands for Multilayer Perceptron. This indicates a channel splicing operation. This indicates a global average pooling operation. This indicates a global max pooling operation.

[0088] (2) To suppress interference from irrelevant regions in the context features, we further utilize a channel attention module to capture channel-level dependencies and important feature correlations between different feature channels. Given input features Features are obtained through global average pooling. Then, a channel attention map is generated using a multilayer perceptron and a sigmoid activation function. Finally, multiplying it with the original image enhances the representation of key features.

[0089]

[0090]

[0091]

[0092] in, Key features of channel attention This represents the sigmoid operation, MLP stands for Multilayer Perceptron, and H and W represent the height and width of the image, respectively. i, j All represent pixels.

[0093] (2) Feature fusion.

[0094] To enable the network to better focus on prominent target regions, we fuse attention features from both spatial and contextual paths. The fused features not only fully utilize spatial feature information but also channel feature information, obtaining category-related spatial contextual features while suppressing redundant features irrelevant to the target. Through the fusion of two-branch networks, the extracted features are more robust and discriminative than those from single-path networks.

[0095]

[0096] in, Characteristics of initial fusion This indicates element-wise addition.

[0097] To improve accuracy without sacrificing speed, we also investigated parallel depthwise convolution (DW) and pointwise convolution (PW). After passing through DW and PW, the fused features not only extract the feature information of each channel in a refined manner, but also preserve the channel relationships between features, which can significantly reduce the number of convolution kernels and computational cost while ensuring classification accuracy.

[0098]

[0099]

[0100]

[0101] in, Features of depthwise convolution i, j This represents the spatial location of the output feature map. s Represents the channel index. m, n Represents the spatial location of the convolution kernel. The weights represent the convolution kernel. For the features of pointwise convolution, Represents the channel index before pointwise convolution. This represents the channel index after pointwise convolution. The weights represent the convolution kernel. This is a characteristic of the final fusion.

[0102] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-branch image classification method based on spatial-context, characterized in that, Specifically, the following steps are included: S1. Data preprocessing: Performing preprocessing operations on the input data; S2. Dataset partitioning: Divide the preprocessed data into training set, validation set and test set according to the proportions; S3. Model Training: Train the entire multi-branch image classification model using the pre-defined training set until the loss converges; Given an input image X, extract features through spatial path and context path respectively and perform feature fusion. S4. Model Testing: Test the classification model using the predefined test set and training weights, and output the results; S5. Model Performance Verification: Calculate the performance evaluation index for the output results. If the performance is good, the training ends; otherwise, modify the model parameters and retrain. In step S3, the context path first uses the ResNet34 model to extract the depth features of the input image, and then uses the global context module to obtain the context features based on the depth features. The spatial path shares the parameters of the first three layers of the ResNet34 model; the formula for the output features extracted by the spatial path is: in, Features extracted for spatial paths; This represents the third layer features of ResNet34; X represents the input image; The spatial path includes a spatial attention module that enables the network to extract key features with spatial location recognition; the context path includes a channel attention module that captures channel-level dependencies and important feature correlations between different feature channels. In step S3, the features after feature fusion undergo depthwise convolution and pointwise convolution; the feature formulas for depthwise convolution and pointwise convolution are as follows: in, i, j This represents the spatial location of the output feature map; Features representing depthwise convolution; s Represents the channel index; m, n Indicates the spatial location of the convolution kernel; The weights represent the convolution kernel; Features of pointwise convolution; Represents the channel index before pointwise convolution; Represents the channel index after pointwise convolution; The weights represent the convolution kernel; This is a characteristic of the final fusion.

2. The multi-branch image classification method based on spatial-context as described in claim 1, characterized in that, The formula for the depth feature is: ； X∈R 224x224x3 ; in, This represents the number of channels in the i-th feature extraction layer of the ResNet34 model. The features extracted by ResNet34 are represented by H, W, and C, which represent the height, width, and number of channels of the image, respectively. R indicates that it is an RGB image, 224 represents the image size, and 3 represents the number of channels.

3. The spatial-context-based multi-branch image classification method according to claim 1, characterized in that, The formula for the global context module is: in, Indicates the global context module; and All are input features; Represents a 1x1 convolution; ReLU represents an activation function; LN represents layer normalization. For contextual features; These are the features of the last layer of ResNet34.

4. The multi-branch image classification method based on spatial-context as described in claim 1, characterized in that: Convolutional and pooling layers are used to extract the deep features required for classification.

5. The spatial-context-based multi-branch image classification method according to claim 1, characterized in that, In the spatial attention module, input features are first given. Two features are obtained using global average pooling and global max pooling. , Spatial attention maps are then generated using a shared multilayer perceptron and sigmoid activation function. Finally, multiply with the original image; where H and W represent the image height and width respectively, and C represents the number of channels. Represents spatial input features, This represents the spatial global average pooling feature. Represents the global max pooling feature in space. Represents a spatial attention map; The The formula The formula The formulas are as follows: In the formula, Key features of spatial attention; This represents the sigmoid operation; MLP stands for Multilayer Perceptron. This indicates a channel splicing operation; This indicates a global average pooling operation; This indicates a global max pooling operation.

6. The spatial-context-based multi-branch image classification method according to claim 5, characterized in that, In the channel attention module, input features are first given. Features are obtained through global average pooling. Subsequently, a channel attention map is generated using a multilayer perceptron and a sigmoid activation function. Finally, multiply by the original image; where, Indicates channel input characteristics, This represents the global average pooling characteristic of the channel. This represents the channel attention map, where H and W represent the height and width of the image, respectively, and C represents the number of channels; The The formula The formulas are as follows: in, Key features representing channel attention; H represents the sigmoid operation; MLP stands for Multilayer Perceptron; H represents the height of the image; W represents the width of the image. i,j Represents pixels; This is a characteristic of initial fusion; This indicates element-wise addition.

Images