Image classification method and system based on space attention and sequence modeling

Abstract

The invention relates to the technical field of computer vision and deep learning, and discloses an image classification system and method fusing a space attention mechanism and long and short-term memory network sequence modeling, which combines the feature extraction capability of a convolutional neural network and space attention and time sequence attention mechanisms. And processing the spatial position sequence by using a long-short-term memory network. Firstly, advanced spatial features of an image are extracted through a feature extraction module by adopting a pre-trained convolutional neural network with a frozen weight, then a spatial attention module is introduced, a spatial attention graph is generated through channel dimension statistics, and important region features are enhanced. After the spatial attention is weighted, a dual-path feature is utilized, one path enters a feature transformation module, and a convolutional layer is used for reducing dimensionality and enhancing feature expression ability. Then, a sequence modeling module is carried out, the spatial features are flattened into a position sequence, a spatial position dependency relationship is modeled by adopting a bidirectional long-short-term memory network, and learnable attention vector dynamic aggregation key position features are introduced; the other path retains spatial global features. And outputting the dual-path features to a feature fusion module, extracting spatial global features and sequence aggregation features in parallel, and designing a gating mechanism to adaptively fuse the dual-path features. And finally, entering a classification module, and realizing end-to-end image classification based on fusion features.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning technology, and specifically to an image classification method and system that combines a spatial attention mechanism with long short-term memory network sequence modeling. The method and system are particularly suitable for fine-grained image classification tasks that require precise spatial understanding, such as medical image analysis, industrial quality inspection, and natural species identification. Background Art

[0002] With the rapid development of computer vision and deep learning, the task of classifying images has become intelligent. The essence of image classification is to allow computers to understand the semantics of image content, classify it into predefined categories, extract features from the image to determine its category, and meet the image recognition and classification needs of users.

[0003] Although high accuracy has been achieved in image classification tasks using convolutional neural networks, existing image classification systems mostly extract features through the local perception characteristics of convolution kernels, making it difficult to model dependencies between long-distance spatial positions. In addition, existing attention methods (such as CBAM) lack the ability to dynamically model spatial sequences to capture complex positional relationships. Existing technologies are mainly aimed at video temporal modeling and lack a dedicated processing architecture for the spatial position sequence of a single image, resulting in limited understanding of complex spatial layouts. Secondly, some global pooling operations will lose key spatial structural information and cannot guarantee fine-grained classification accuracy. Therefore, solving the shortcomings of existing image classification methods in spatial relationship modeling, feature retention capabilities, and adaptive fusion, and providing higher-precision fine-grained classification has become an urgent problem to be solved in this field. Summary of the Invention

[0004] Purpose of the invention: To address the above problems, the present invention proposes an image classification method and system based on spatial attention and sequence modeling.

[0005] In order to solve the above problems, the present invention adopts the following technical solutions:

[0006] An image classification method based on spatial attention and sequence modeling, characterized by comprising the following steps:

[0007] S1, extracts the spatial feature map of the input image through a pre-trained convolutional neural network;

[0008] S2, applying a spatial attention mechanism to perform weighted processing on the spatial feature map to generate an attention-enhanced feature map;

[0009] S3, flattening the attention-enhanced feature map into a feature sequence along the spatial dimension;

[0010] S4, using a bidirectional LSTM network to process the feature sequence to obtain sequence modeling features;

[0011] S5, aggregating the sequence modeling features through the attention mechanism to obtain sequence aggregation features;

[0012] S6, extracting spatial global features from the attention-enhanced feature map;

[0013] S7, using a gated fusion mechanism to dynamically fuse the sequence aggregation features and spatial global features;

[0014] S8, performing image classification based on the fused features.

[0015] An image classification system based on spatial attention and sequence modeling, characterized by comprising:

[0016] Feature extraction module;

[0017] Spatial attention module;

[0018] Sequence modeling module;

[0019] Feature fusion module;

[0020] Classification module.

[0021] Optionally, the S1 includes:

[0022] Use ResNet-50 network for feature extraction;

[0023] The network parameters remain frozen during training.

[0024] Optionally, the S2 includes:

[0025] Calculate the average and maximum values ​​of the spatial feature map in the channel dimension;

[0026] The average feature map and the maximum feature map are concatenated, and spatial attention weights are generated through a convolutional layer; the attention weights are normalized using a Sigmoid function.

[0027] Optionally, the S3 includes:

[0028] Compress the channel dimension through the convolution layer to perform feature dimensionality reduction transformation;

[0029] Convert the two-dimensional feature map into a spatial sequence.

[0030] Optionally, the S4 includes:

[0031] The bidirectional LSTM network has a multi-layer structure;

[0032] The hidden layer size is configurable between 64-256;

[0033] A bidirectional connection method is used to process spatial sequences.

[0034] Optionally, the S5 includes:

[0035] Compute sequence position importance scores using learnable query vectors;

[0036] The attention weight is obtained by normalizing the Softmax function, and then the sequence features are weighted and summed based on the attention weight.

[0037] Optionally, the S6 includes:

[0038] The attention-enhanced feature map is globally average pooled and the feature vector after global average pooling is transformed through a fully connected layer to generate global spatial features.

[0039] Optionally, the S7 includes:

[0040] Concatenate spatial global features and sequence aggregation features and generate fusion weights through fully connected layers and Sigmoid functions;

[0041] The final feature is calculated according to the formula fusion feature = gating weight × spatial global feature + (1-gating weight) × sequence aggregation feature.

[0042] Optionally, the image classification system based on spatial attention and sequence modeling is characterized by comprising:

[0043] The feature extraction module includes a ResNet-50 network with frozen parameters, outputting a 7×7×2048 spatial feature map;

[0044] The spatial attention module includes generating feature maps through dual-path pooling (channel mean pooling and maximum pooling), and outputting spatial attention weights through convolution and Sigmoid normalization;

[0045] Sequence Modeling Module: This module implements feature dimensionality reduction and serialization conversion through convolutional layers, and uses a bidirectional LSTM with configurable hidden layer dimensions (64-256) to process spatial sequences.

[0046] The feature fusion module comprises fusing the aggregated features output by the sequence modeling module with the spatial global features;

[0047] The classification module includes a classification module: performing full connection transformation and ReLU activation on the fusion features, and outputting the classification results.

[0048] Compared with the prior art, the advantages of the present invention are:

[0049] 1. This invention reconstructs a two-dimensional feature map into a sequence of spatial positions and models long-range dependencies between spatial positions using a bidirectional LSTM network. Compared to the local receptive field limitations of traditional CNNs, this method can establish direct correlations between any two spatial positions. For example, in medical image analysis, it can simultaneously capture the spatial distribution patterns between dispersed microcalcification lesions; in industrial quality inspection scenarios, it can identify the topological correlation between surface scratches and internal cracks on parts. This global spatial modeling capability solves the problem of feature fragmentation in fine-grained classification, enabling the model to understand the complex spatial layout of object components with unprecedented accuracy.

[0050] 2. This invention introduces spatial and temporal attention, a dual-dimensional attention architecture that integrates spatial and temporal attention, innovatively integrating the temporal attention mechanism with the traditional CBAM spatial attention. The spatial attention module generates a spatial weight map through dual-path aggregation of the mean and maximum values ​​of the channel dimensions, precisely focusing on key areas. Temporal attention dynamically adjusts the contribution weight of each time step in the LSTM sequence through a learnable scaling parameter, achieving adaptive optimization of feature selection. The synergy of these two significantly improves the model's ability to discriminate complex visual features.

[0051] 3. This invention adopts dual optimization of parameter freezing and feature dimensionality reduction to freeze the ResNet50 backbone network parameters to avoid large-scale parameter updates. The number of channels is compressed from 2048 to 512 through 1×1 convolution, and the LSTM input sequence length is compressed from 7×7=49 dimensions to the flattened dimension of the feature map, significantly reducing the complexity of time series processing.

[0052] 4. In the feature fusion module, the present invention designs a gated adaptive fusion module to break through the limitations of traditional feature splicing. The spatial branch extracts high-level semantic features through global pooling, retaining the deep representation of ResNet50. The temporal branch uses bidirectional LSTM to parse the contextual dependency of the spatial sequence and dynamically learns the fusion weights of spatial and temporal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention.

[0055] Figure 2 A module diagram of an image classification system based on spatial attention and sequence modeling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0057] The present invention is an image classification method and system based on spatial attention and sequence modeling. The present invention will be described in more detail below with reference to the accompanying drawings and technical solutions. The image classification method process and system module diagram based on spatial attention and sequence modeling are shown in FIG. Figure 1 , as shown in 2, including the following steps:

[0058] S1, extracts the spatial feature map of the input image through a pre-trained convolutional neural network;

[0059] S2, applying a spatial attention mechanism to perform weighted processing on the spatial feature map to generate an attention-enhanced feature map;

[0060] S3, flattening the attention-enhanced feature map into a feature sequence along the spatial dimension;

[0061] S4, using a bidirectional LSTM network to process the feature sequence to obtain sequence modeling features;

[0062] S5, aggregating the sequence modeling features through the attention mechanism to obtain sequence aggregation features;

[0063] S6, extracting spatial global features from the attention-enhanced feature map;

[0064] S7, using a gated fusion mechanism to dynamically fuse the sequence aggregation features and spatial global features;

[0065] S8, performing image classification based on the fused features.

[0066] S1 includes:

[0067] Use ResNet-50 network for feature extraction;

[0068] The network parameters remain frozen during training.

[0069] Extracts multi-level spatial features from the input image. ResNet-50 is a deep residual network with 50 layers. By introducing residual connections, it effectively alleviates the vanishing gradient problem in deep neural networks and enhances feature extraction capabilities. This network structure is suitable for processing multi-scale images and can extract multi-level, multi-dimensional image features from images. It serves as the feature extraction backbone and outputs a high-dimensional spatial feature map (2048 channels, 7×7 spatial size). This corresponds to the feature extraction module described in the module diagram of an image classification system based on spatial attention and sequence modeling, providing the foundation for subsequent attention mechanisms and sequence modeling.

[0070] Its network structure is:

[0071] Input layer: Accepts a preprocessed RGB image (224×224). The input image is normalized and the pixel values ​​are mapped to the range [0, 1].

[0072] Convolutional layer: ResNet-50 consists of a convolutional layer that applies 64 7×7 convolution kernels with a stride of 2, followed by max pooling, and then outputs to four convolutional layers. Each convolutional layer consists of 3 residual blocks, 4 residual blocks, 6 residual blocks, and 3 residual blocks, respectively, forming a 50-layer network architecture. The output (i, j) of the first convolutional layer is:

[0073]

[0074] where K (m，n) is the weight of the convolution kernel at (m, n), where the convolution kernel size is 7×7. (i·2+m，j·2+n) is the pixel value of the input image at coordinates (i·2+m, j·2+n) (convolution sampling with a stride of 2), b is the bias term, and i, j are the indices of the output image.

[0075] Residual block: The core structure of ResNet-50 is residual learning. By using skip connections to add the input directly to the output, the gradient can be directly back-propagated to the previous layer to solve the gradient disappearance problem in deep networks. The calculation formula is:

[0076] y=F(x,{Wi})+x (2)

[0077] Where x is the input feature, typically the output of the previous layer. F(x, {Wi}) is the residual function, which represents the transformation of the input x by one or more convolutional layers. {Wi} is the learnable parameter contained in this function, namely the convolution kernel weight. y is the output feature, which is obtained by adding the output of the residual function to the original input.

[0078] Pooling layer: Between every two convolutional layers, a maximum pooling layer is inserted to reduce the spatial size of the feature map while retaining the most important feature information. The maximum pooling operation is performed by the following formula:

[0079] MaxPool=max(x i，j ) (3)

[0080] Among them, x i，j To pool the pixel values ​​in the area, a 3×3 sliding window slides on the input feature map with a step size of 2, covering a 3×3 area each time.

[0081] Fully connected layer: The fully connected layer here is actually the output layer and is not included in the 50-layer structure.

[0082] S2 includes:

[0083] Calculate the average and maximum values ​​of the spatial feature map in the channel dimension. The specific operations are:

[0084] Calculate the average value of each spatial position over all channels to reflect the global feature response of that position.

[0085] The maximum value of each spatial position in all channels is extracted to reflect the most significant feature of that position.

[0086] It is done by the following formula:

[0087]

[0088] Among them, b is the batch index, c is the channel dimension, and (h, w) is the spatial position. The value of the feature map output by ResNet-50 at batch b, channel c, position (h, w). is the mean feature map of the channel dimension. Maximum feature map of the channel dimension.

[0089] The average feature map and the maximum feature map are concatenated and the spatial attention weight is generated through the convolution layer. The specific operation is:

[0090] The results of average pooling and maximum pooling are spliced ​​in dual-path channel dimensions to form a channel feature map. The formula is:

[0091] F combined =[F avg ; F max ] (6)

[0092] The spatial attention weight guides the model to focus on the areas in the image that are more important to the current task. The specific weight generation operation is performed by the following formula:

[0093]

[0094] Among them, W conv is the convolution kernel weight, F combined The value of the concatenated feature map in batch b, channel i, and offset position. k x , k y is the convolution kernel offset, and σ is the Sigmoid function.

[0095] Applying the Sigmoid function to normalize the attention weights will map the convolution output to the interval [0, 1] to generate a spatial attention map.

[0096] The above-mentioned S2 operation corresponds to the spatial attention module described in the module diagram of an image classification system based on spatial attention and sequence modeling.

[0097] S3 includes:

[0098] Compress the channel dimension through the convolution layer to perform feature dimensionality reduction transformation;

[0099] After step S2, the dual-path outputs of S3 and S6 are performed. The feature dimensionality reduction transformation performed by compressing the channel dimension through the convolution layer is to sum the convolution results of all input channels for each output channel to obtain the channel feature map. Batch normalization is then used to accelerate training and improve model stability. Finally, a nonlinear activation function is introduced to enhance the model's expressiveness. The specific operation is performed using the following formula:

[0100]

[0101] Among them, F trans is the feature map after dimensionality reduction, ReLU is the linear rectification activation function, BN is the batch normalization operation, W feat is the convolution kernel weight, F weighted is the weighted feature map, and * is the convolution operation.

[0102] Convert the two-dimensional feature map into a spatial sequence. Spatial serialization is the key to connecting convolutional neural networks (spatial feature extraction) and recurrent neural networks (sequence dependency modeling). The formula is:

[0103]

[0104] Where t=(h t -1)×W+w t , W is the feature map width. In the corresponding formula (8), S is the output serialized feature, b, t, d are the batch, sequence position, and kernel feature dimension index respectively. h t and w t are the original row and column coordinates.

[0105] S4 includes:

[0106] The feature dimensionality reduction performed in S3 is then input into S4. The feature dimensionality reduction described in S2 is completed before the LSTM operation, reducing computational complexity. The bidirectional LSTM network described in S4 is a multi-layered structure, employing a vertically stacked multi-layer LSTM architecture. Each LSTM layer processes the hidden state output by the previous layer, with the bottom layer inputting spatial sequence features and the top layer outputting deep spatial dependency features.

[0107] The hidden layer size is configurable between 64 and 256, and the hidden layer dimension is freely configurable between 64 and 256. By adjusting this parameter, you can control the model capacity and computational complexity, and configure different hidden layer dimensions in different scenarios.

[0108] A bidirectional connection is used to process spatial sequences. The forward LSTM processes the sequence from left to right, and the reverse LSTM processes the sequence from right to left, which can learn long-range dependencies of the data.

[0109] S5 includes:

[0110] The sequence position importance score is calculated using a learnable query vector, which is initialized to a random vector. The training process automatically learns to focus on key feature patterns. After the sequence features are input, the bidirectional LSTM is fed into the output. The position importance is calculated using the learnable query vector and a scaling factor. This is done using the following formula:

[0111] s b，t =H b,t ·(αq) (10)

[0112] Among them H b，t is the bidirectional LSTM position feature vector, α is the scaling factor, and q is the base query vector.

[0113] Attention weights are obtained by normalizing with the Softmax function, and then the sequence features are weighted and summed based on the attention weights. The importance scores are mapped to a probability distribution with a sum of 1. The attention weights represent the relative importance of features at different positions in the sequence. Then, a weighted sum is performed, and features are screened based on the weights, with higher importance being given greater weights. This is done using the following formula:

[0114]

[0115] Formula (11) is the attention weight operation, and formula (12) is the weighted sum of attention weights. b，t is the position weight, T is the sequence length, such as T is 49 after the 7×7 feature map is flattened, f temporal It is a weighted sum of sequence aggregation features.

[0116] The above S3, S4, and S5 correspond to the sequence modeling module described in the module diagram of an image classification system based on spatial attention and sequence modeling.

[0117] S6 includes:

[0118] Global average pooling is performed on the attention-enhanced feature map, and the feature vector after global average pooling is transformed through a fully connected layer to output global spatial features. The feature map after spatial attention weighting is received, and the feature response of the key areas has been enhanced by step S2. The feature tensor output by global average pooling is flattened and output through a fully connected layer, and the channel dimension is aligned with the output of step S5.

[0119] S7 includes:

[0120] The spatial global features and sequence aggregate features are concatenated and fusion weights are generated using a fully connected layer and a sigmoid function. The sequence aggregate features of S5 are concatenated with the spatial global features of S6, and the two 256-dimensional feature vectors are concatenated end to end. This corresponds to the feature fusion module described in the module diagram of an image classification system based on spatial attention and sequence modeling. This operation is performed using the following formula:

[0121] f fusion =g·f spatial +(1-g)·f temporal (13)

[0122] Among them, g is the spatial feature weight, 1-g is the temporal feature weight, and f spatial is the global feature of space, f temporal is the sequence aggregation feature.

[0123] The S8 includes:

[0124] The fused features from S7 are received and transformed in the fully connected layer, while maintaining the same dimension. The nonlinear transformation is then applied to the ReLU activation function to enhance the expressive power of the features. The final output is used for image classification. This corresponds to the classification module described in the module diagram of an image classification system based on spatial attention and sequence modeling.

[0125] Through experiments, 12,500 images in the "Cat and Dog Dataset" were classified, and the average confidence level in the unlabeled test set was 99.31%, of which the high confidence ratio was 97.78%.

[0126] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are provided in the following preferred embodiments. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0127] The above description is only a preferred embodiment of the present invention. The technical features of the above embodiments can be combined arbitrarily (as long as there is no contradiction in the combination of these technical features). In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

Claims

1. An image classification method based on spatial attention and sequence modeling, characterized in that: The following steps are involved: S1, extracts the spatial feature map of the input image through a pre-trained convolutional neural network; S2, applying a spatial attention mechanism to perform weighted processing on the spatial feature map to generate an attention-enhanced feature map; S3, flattening the attention-enhanced feature map into a feature sequence along the spatial dimension; S4, using a bidirectional LSTM network to process the feature sequence to obtain sequence modeling features; S5, aggregating the sequence modeling features through the attention mechanism to obtain sequence aggregation features; S6, extracting spatial global features from the attention-enhanced feature map; S7, using a gated fusion mechanism to dynamically fuse the sequence aggregation features and spatial global features; S8, performing image classification based on the fused features.

2. An image classification system based on spatial attention and sequence modeling, characterized in that: include: The feature extraction module consists of a pre-trained residual neural network to extract the spatial features of the input image; The spatial attention module consists of a channel statistics unit and a convolutional layer to generate a spatial attention map and weight the spatial feature map; Sequence modeling module, which includes a bidirectional LSTM network and a sequence attention unit to process sequence features; Feature fusion module, with a learnable gating mechanism, for dynamically fusing spatial global features and sequence features; The classification module is used to output classification results based on the fusion features.

3. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that In S1: Use ResNet50 network as convolutional neural network; The network parameters remain frozen during training.

4. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that The S2 includes: Calculate the average and maximum values ​​of the spatial feature map in the channel dimension; splicing the average value feature map and the maximum value feature map; Generate spatial attention weights through convolutional layers; Apply the Sigmoid function to normalize the attention weights.

5. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that In the S3: Compress the channel dimension through the convolution layer to perform feature dimensionality reduction transformation; Convert the two-dimensional feature map into a spatial sequence.

6. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that In said S4: The bidirectional LSTM network has a multi-layer structure; The hidden layer size is configurable between 64-256; A bidirectional connection method is used to process spatial sequences.

7. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that The S5 includes: Compute sequence position importance scores using learnable query vectors; The attention weight is obtained by normalizing the Softmax function; Weighted summation of sequence features based on attention weights.

8. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that The S6 includes: The attention-enhanced feature map is globally average pooled and the feature vector after global average pooling is transformed through a fully connected layer to generate spatial global features.

9. The image classification method based on spatial attention and sequence modeling according to claim 1, characterized in that The S7 includes: Splicing spatial global features and sequence aggregation features; Generate fusion weights through the fully connected layer and Sigmoid function; The final feature is calculated according to the formula fusion feature = gating weight × spatial global feature + (1-gating weight) × sequence aggregation feature.

10. The image classification system based on spatial attention and sequence modeling according to claim 2, characterized in that include: Feature extraction module: uses a ResNet-50 network with frozen parameters to output a 7×7×2048 spatial feature map; Spatial attention module: Generates feature maps through dual-path pooling (channel mean pooling and maximum pooling), and outputs spatial attention weights through convolution and sigmoid normalization; Sequence Modeling Module: This module implements feature dimensionality reduction and serialization conversion through convolutional layers, and uses a bidirectional LSTM with configurable hidden layer dimensions (64-256) to process spatial sequences. Feature fusion module: fuses the aggregated features output by the sequence modeling module with the spatial global features; Classification module: performs full connection transformation and ReLU activation on the fused features and outputs the classification results.

Images