Image recognition method based on multi-scale feature fusion

Abstract

The invention provides an image recognition method based on multi-scale feature fusion, and relates to the technical field of computer vision and image recognition. And a recognition result is output by constructing a multi-scale image recognition model. According to the method, a feature coding module is adopted to extract multi-scale features, based on a parallel local and global feature extraction mechanism, local detail features are extracted through a local feature extraction module, and global context information is captured through a global feature extraction module; a cross-branch fusion module is designed, and adaptive fusion of local and global features is realized through three-stage processing of channel attention, feature re-calibration and residual gating fusion; a multi-branch deep interaction module is introduced, and multilevel spatial features are captured through a parallel path of a differentiated receptive field; and in cooperation with a lightweight classification module, excellent classification performance is obtained while the calculation efficiency is ensured, and reliable technical support is provided for related intelligent services.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image recognition, and in particular to an image recognition method based on multi-scale feature fusion. Background Art

[0002] With the development of deep learning, traditional convolutional neural network (CNN) methods have been widely used in image classification tasks. However, convolution-based models have inherent limitations in modeling long-range feature dependencies. They mainly expand the receptive field by stacking convolutional layers, which increases the model depth and consumes significant computing resources. At the same time, it is difficult to capture a wide range of global semantic features. In addition, convolutional neural networks usually have difficulty adaptively adjusting the feature focus range according to the key parts of the target object, and are prone to ignoring local discriminative features, which in turn affects classification performance. To address the above problems, there is an urgent need for a new image classification method that can simultaneously take into account local feature modeling and long-range dependency capture to improve the accuracy of image recognition and the overall robustness of the model. Summary of the Invention

[0003] In response to the problems in the prior art of insufficient utilization of multi-scale structural information and limited local and global feature expression capabilities in image recognition, the present invention proposes an image recognition method based on multi-scale feature fusion. First, a feature encoding module is used to extract multi-scale feature maps of the image, and the multi-scale feature maps are mapped to the same spatial resolution and then fused; the fused features are respectively input into the local feature extraction module and the global feature extraction module to fully mine key classification information from different receptive field levels; the local and global features are interactively fused through the cross-branch fusion module, and a multi-branch deep interaction module is introduced to enhance the feature expression capability, thereby improving the discrimination accuracy and generalization robustness of the model in image classification tasks.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] The present invention provides an image recognition method based on multi-scale feature fusion, comprising the following steps:

[0006] S1, obtain original image data, perform preprocessing and enhancement on the obtained original image data, and construct a classification data set; divide the classification data set into a training set, a validation set, and a test set in proportion;

[0007] The preprocessing includes: unifying the size of the original image obtained, performing standardization processing, center cropping and boundary clearing on the original image after the size is unified, and obtaining the preprocessed original image;

[0008] The enhancement processing includes: enhancing the pre-processed original image by using random rotation, flipping and brightness perturbation methods;

[0009] S2, builds a multi-scale image recognition model, including: feature encoding module, local feature extraction module, global feature extraction module, cross-branch fusion module, multi-branch deep interaction module and classification module;

[0010] For the original image input into the multi-scale image recognition model, the feature encoding module is used to extract the multi-scale feature map that is refined layer by layer, and the multi-scale feature map is unified to the smallest spatial size among all scales. The scale-aligned feature map is obtained by channel splicing; the scale-aligned feature map is transmitted to the local feature extraction module and the global feature extraction module respectively to extract the local feature map and the global feature map respectively; the local feature map and the global feature map are passed to the cross-branch fusion module for fusion to generate a fused feature map, which is transmitted to the multi-branch deep interaction module to further optimize the fused feature map. The optimized feature map is input into the classification module to obtain the classification result of the original image;

[0011] S21, use the Swin-Transformer network to build a feature encoding module, use the feature encoding module to perform multi-scale feature extraction, scale alignment and feature splicing on the input original image to obtain a scale-aligned feature map;

[0012] The feature encoding module is constructed using the Swin-Transformer network. The Swin-Transformer network includes multiple basic encoding units and scale alignment and feature splicing units. Each basic encoding unit includes a patch merging submodule, a local window self-attention submodule, and a sliding window self-attention submodule. The scale alignment and feature splicing unit includes a pooling downsampling layer and a splicing layer.

[0013] Use the feature encoding module to perform multi-scale feature extraction, scale alignment, and feature splicing on the input original image to obtain a scale-aligned feature map. The specific method is as follows:

[0014] In the Patch Merging submodule, the original image input to the Patch Merging submodule is divided into 4×4 regions. Four spatially adjacent 2×2 regions are divided into a patch. The feature vectors of each region in the patch are extracted. The feature vectors of these four regions are concatenated in the channel dimension and then linearly transformed to obtain a feature map with half the width and height and double the number of channels.

[0015] In the local window self-attention submodule, for a feature map X∈R with C channels and H×W size H×W×C , divide the feature map X into N local windows of size M×M, perform single-head self-attention calculation on each local window, and the single-head self-attention head of the nth local window is n; Concatenate and linearly project the single-head attentions of N local windows to obtain a multi-head self-attention feature map;

[0016] In the sliding window self-attention submodule, the sliding window size is set to M×M, and the initial position of the sliding window is offset by a fixed step size relative to the sliding window position of the sliding window self-attention submodule in the previous basic coding unit. The calculation method of the sliding window self-attention feature map is the same as the attention method within the local window, only the input window area is different. The feature map X with C channels and size H×W passes through the local window self-attention submodule and the sliding window self-attention submodule in sequence to obtain the feature map output by the basic coding unit.

[0017] For the multiple basic coding units of the Swin-Transformer network, which output multiple feature maps of different scales, the feature map with the smallest scale is used as the baseline. The pooling downsampling layer is used to perform pooling downsampling operations on the feature maps of the remaining scales to keep the scales of the feature maps output by the multiple basic coding units consistent. The splicing layer is then used to splice the feature maps with the same scale in the channel dimension to obtain the scale-aligned feature map F. aligned ;

[0018] S22, constructing a local feature extraction module, including multiple parallel branches and a local enhancement submodule; using the local feature extraction module to process the scale-aligned feature map to obtain a local feature map;

[0019] The local feature extraction module contains five parallel branches, specifically:

[0020] The first branch includes 3×3 depth-wise separable convolution and 1×1 convolution;

[0021] The second branch includes 3×3 convolution, SE attention layer and 1×1 convolution;

[0022] The third branch includes 1×3 convolution, 3×1 convolution, coordinate convolution CoordConv and 1×1 convolution;

[0023] The fourth branch includes 3×3 maximum pooling, 3×3 convolution and 1×1 convolution;

[0024] The fifth branch includes a 3×3 dilated convolution with a dilation rate of 2, a ReLU activation function, and a 1×1 convolution;

[0025] The local enhancement submodule includes a 3×3 convolution, a Softmax normalization layer, and a weighted fusion layer;

[0026] The five branches of the local feature extraction module use different types of convolution to align the scale feature map F alignedProcessing is performed, the feature maps output by each branch are spliced ​​in the channel dimension, and the spliced ​​feature maps are optimized through the local enhancement submodule to obtain the local feature map F local ;

[0027] S23, constructing a global feature extraction module, including a channel attention submodule, a context feature extraction submodule, and residual enhancement; using the global feature extraction module to process the scale-aligned feature map to obtain a global feature map;

[0028] In the channel attention submodule, the scale-aligned feature map F aligned Perform global average pooling to obtain the global description vector of each channel, and construct the channel attention weight through the fully connected mapping layer. Multiply the channel attention weight with the feature map aligned with the scale of the input channel attention submodule by element according to the channel dimension to generate the weighted feature map X attn ;

[0029] The weighted feature map X attn The input context feature extraction submodule is flattened into a one-dimensional vector, and the hidden dimension is compressed and restored through the multi-layer perceptron to obtain the context enhanced feature map X gce ;

[0030] The weighted feature map X is enhanced by residual attn and context-enhanced feature map X gce Perform element-by-element residual connection to obtain the global feature map F global ;

[0031] S24, constructing a cross-branch fusion module, including a channel attention submodule, a feature recalibration submodule, and a residual gated fusion submodule; inputting local features and global features into the cross-branch fusion module for adaptive fusion to obtain adaptively fused features;

[0032] The channel attention submodule is based on the global feature map F global Generate channel attention weight α local , using channel attention weight α local For the local feature map F local Perform element-by-element multiplication to obtain the enhanced local feature map F′ local ;

[0033] The feature recalibration submodule sequentially recalibrates the enhanced local feature map F′ local and the global feature map F global Processing, specifically:

[0034] Use 1×1 convolution to enhance the local feature map F′ local and the global feature map F globalAdjust the number of channels, use the splicing operation to fuse the adjusted local feature map and the global feature map in the channel dimension, use 3×3 convolution to extract spatial features from the fused feature map, and obtain the recalibrated feature map F through the BN layer and ReLU activation function. concat ;

[0035] The residual gated fusion submodule balances the contribution of the recalibrated features and the enhanced local features through the gating mechanism to obtain the adaptively fused feature map F fused ;

[0036] S25: Construct a multi-branch deep interaction module, including multiple parallel paths, a concatenation layer, and a 1×1 convolution layer. Input the adaptively fused features into the multi-branch deep interaction module, use multiple parallel paths to extract multi-scale features under different receptive fields, concatenate the output features of each path in the channel dimension, and then perform channel compression and feature fusion through 1×1 convolution to obtain the feature map output by the multi-branch deep interaction module.

[0037] The multi-branch deep interaction module contains three parallel paths, specifically:

[0038] The first path consists of two serial 3×3 convolutions;

[0039] The second path includes a 5×5 convolution;

[0040] The third path includes a 3×3 max pooling and a 1×1 convolution;

[0041] S26, uses the MLP network to build a classification module, including a global average pooling layer, two fully connected layers, an intermediate batch normalization layer, and a ReLU activation layer; the features output by the multi-branch deep interaction module are input into the classification module and the prediction results are output;

[0042] The feature map output by the multi-branch deep interaction module is input into the classification module, and the channel feature vector is obtained through global average pooling. The channel feature vector is reduced to the intermediate feature dimension through the first fully connected layer, and the predicted classification result is output in sequence through the intermediate batch normalization layer, the ReLU activation layer, and the second fully connected layer.

[0043] S3, use the training set to supervise the multi-scale image recognition model. After the training is completed, the obtained model is verified using the validation set, and the model with the best performance on the validation set is selected as the trained multi-scale image recognition model;

[0044] Construct an auxiliary classification module with the same structure as the classification module, apply the auxiliary classification module to the output feature map of the local feature extraction module, and use the auxiliary classification module to provide additional supervision. Both the auxiliary classification module and the classification module use the standard cross entropy loss function.

[0045] The multi-scale image recognition model is trained using the training set, employing a fully supervised learning approach. A joint loss function for the multi-scale image recognition model is constructed based on the loss functions of the classification module and the auxiliary classification module. A weighted loss function strategy is employed to dynamically assign weights to different categories, assigning higher weights to categories that appear fewer than a set threshold during training. End-to-end training is completed using a backpropagation mechanism. After training, the resulting model is validated using the validation set, and the model with the best performance on the validation set is selected as the final multi-scale image recognition model.

[0046] S4, after uniform preprocessing of the image to be classified, input it into the trained multi-scale image recognition model and output the recognition result;

[0047] The beneficial effects of the above technical solution are as follows: the present invention provides an image recognition method based on multi-scale feature fusion, which uses a feature encoding module to extract multi-scale features and construct four feature scales. While keeping the number of channels unchanged, the first three scales are unified by pooling operations, aligned with the fourth scale features, and then spliced ​​in the channel dimension, effectively integrating spatial information of different scales and improving feature description capabilities; based on a parallel local-global feature extraction mechanism, the local feature extraction module extracts local detail features, while the global feature extraction module captures global context information. The two features complement each other and enhance each other, significantly improving the model's recognition ability for key parts; a cross-branch fusion module is designed, which achieves adaptive fusion of local and global features through a three-stage process of channel attention, feature recalibration, and residual gated fusion, thereby improving the discriminability and robustness of feature expression; a multi-branch deep interaction module is introduced to capture multi-level spatial features through parallel paths of differentiated receptive fields, and combined with a lightweight MLP classification head design, excellent classification performance is achieved while ensuring computational efficiency; the method of the present invention can effectively fuse multi-scale visual features, improve the accuracy and generalization ability of image classification, have broad application prospects, and provide reliable technical support for related intelligent services. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of an image recognition method based on multi-scale feature fusion provided by an embodiment of the present invention;

[0049] Figure 2 A structural diagram of a multi-scale image recognition model provided by an embodiment of the present invention;

[0050] Figure 3 A diagram of the Swin Transformer network structure provided by an embodiment of the present invention;

[0051] Figure 4 A structural diagram of a local feature extraction module provided in an embodiment of the present invention;

[0052] Figure 5 A structural diagram of a global feature extraction module provided in an embodiment of the present invention;

[0053] Figure 6 A structural diagram of a cross-branch fusion module provided in an embodiment of the present invention;

[0054] Figure 7 This is a structural diagram of the multi-branch deep interaction module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0056] This embodiment provides an image recognition method based on multi-scale feature fusion, such as Figure 1 As shown, the following steps are included:

[0057] S1, obtain original image data, perform preprocessing and enhancement on the obtained original image data, and construct a classification data set; divide the classification data set into a training set, a validation set, and a test set in proportion;

[0058] The preprocessing includes: unifying the size of the acquired original image, performing image standardization, center cropping and boundary clearing on the original image after the size is unified, to obtain the preprocessed original image;

[0059] The enhancement processing includes: enhancing the pre-processed original image by using random rotation, flipping and brightness perturbation methods;

[0060] In this embodiment, RGB original image data with clear category labels is obtained, and the size of each obtained RGB original image is uniformly adjusted to 224×224. After adjustment, the size of each original image is [3,224,224], where 3 represents the three RGB channels and 224×224 is the adjusted image size.

[0061] The resized original images were normalized to the range [0, 1], and redundant background was removed through center cropping and boundary cleaning. To improve generalization performance, the preprocessed original images were enhanced using random rotation, flipping, and brightness perturbation methods. A classification dataset was constructed and divided into training, validation, and test sets in an 8:1:1 ratio.

[0062] Use data augmentation operations in the training set to improve image diversity, including color perturbation, cropping, and scaling operations to improve image diversity;

[0063] S2, builds a multi-scale image recognition model, including: feature encoding module, local feature extraction module, global feature extraction module, cross-branch fusion module, multi-branch deep interaction module and classification module;

[0064] Multi-scale image recognition models such as Figure 2 As shown in the figure, for the original image input into the multi-scale image recognition model, the feature encoding module is used to extract the multi-scale feature map that is refined layer by layer, the multi-scale feature map is unified to the smallest spatial size among all scales, and the scale-aligned feature map is obtained by channel splicing; the scale-aligned feature map is transmitted to the local feature extraction module and the global feature extraction module respectively, and the local feature map and the global feature map are extracted respectively; the local feature map and the global feature map are passed to the cross-branch fusion module for fusion to generate a fused feature map, and the fused feature map is transmitted to the multi-branch deep interaction module to further optimize the fused feature map, and the optimized feature map is input into the classification module to obtain the classification result of the original image.

[0065] S21 uses the Swin-Transformer network to construct a feature encoding module, which performs multi-scale feature extraction, scale alignment, and feature splicing on the input original image to obtain a scale-aligned feature map;

[0066] The Swin-Transformer network consists of multiple basic encoding units and scale alignment and feature splicing units. Each basic encoding unit includes a Patch Merging submodule, a local window self-attention submodule, and a sliding window self-attention submodule. The scale alignment and feature splicing unit includes a pooling downsampling layer and a splicing layer.

[0067] The original image input to the feature encoding module passes through multiple basic encoding units in sequence, and the spatial resolution is gradually reduced at different scales through the Patch Merging submodule. The image features are gradually refined through the local window self-attention submodule and the sliding window self-attention submodule, and feature maps with different spatial scales are output;

[0068] The feature encoding module is constructed using the Swin-Transformer network. The Swin-Transformer network structure is as follows: Figure 3 As shown in the figure, it is used to extract multi-scale feature maps of the original image; the feature encoding module built using the Swin-Transformer network has multi-level expression capabilities, gradually compressing the spatial size of the original image and expanding the channel dimension in local modeling and cross-region interaction, while capturing the local details and global context information of the original image. Specifically:

[0069] In the Patch Merging submodule, the original image input to the Patch Merging submodule is divided into 4×4 regions. Four spatially adjacent 2×2 regions are divided into a patch. The feature vectors of each region in the patch are extracted. The feature vectors of these four regions are concatenated in the channel dimension and then linearly transformed to obtain a feature map with half the width and height and double the number of channels.

[0070] In the local window self-attention submodule, for a feature map X∈R with C channels and H×W size H×W×C , divide the feature map X into N local windows of size M×M, perform single-head self-attention calculation on each local window, and the single-head self-attention head of the nth local window is n , as shown in the following formula:

[0071] Q=XW Q ,K=XW K ,V=XW V

[0072]

[0073] Among them, W Q 、W K 、W V is a learnable linear transformation weight matrix, Q, K, and V are query matrix, key matrix, and value matrix respectively, d k is the dimension of the key vector, B rel is a relative position encoding matrix for capturing the relative position between the query matrix and the key matrix;

[0074] The single-head attention of N local windows is spliced ​​and linearly projected to obtain the multi-head self-attention Multihead(Q,K,V), as shown in the following formula:

[0075] Multihead(Q,K,V)=Concat(head1,…,head n ,…,head N )W O

[0076] Among them, Concat(·) is the concatenation operation, W O is the output linear transformation matrix;

[0077] In the sliding window self-attention submodule, the sliding window size is set to M×M. The initial position of the sliding window is offset by a fixed step size relative to the sliding window position of the sliding window self-attention submodule in the previous basic coding unit. The features between the local windows of multiple basic coding units are used to establish cross-region dependencies. The calculation method of the sliding window self-attention is the same as the attention method within the local window, only the input window area is different. The feature map X with C channels and size H×W passes through the local window self-attention submodule and the sliding window self-attention submodule in sequence to obtain the feature map output by the basic coding unit.

[0078] For the multiple basic coding units of the Swin-Transformer network, multiple feature maps of different scales are output. The feature map with the smallest scale is used as the baseline. The pooling downsampling layer is used to perform pooling downsampling operations on the feature maps of other scales to keep the scales of the feature maps output by the multiple basic coding units consistent. The splicing layer is used to splice the feature maps with the same scale in the channel dimension to obtain the scale-aligned feature map Faligned, as shown in the following formula:

[0079] Faligned=Concat(Pool(F1),Pool(F2),…,Pool(FS))

[0080] Where Pool(·) is the pooling downsampling operation, F1, F2, …, FS are the 1st, 2nd, …, Sth basic coding units, and S is the number of basic coding units;

[0081] In this embodiment, the feature encoding module constructed using the Swin-Transformer network includes four basic encoding units. For an RGB original image with an initialization size of [3, 224, 224], after the four basic encoding units, multi-scale feature maps with sizes of [96, 56, 56], [192, 28, 28], [384, 14, 14], and [768, 7, 7] are generated respectively.

[0082] In this example, the scale alignment and feature concatenation module downsamples the first three scale feature maps output by the Swin-Transformer feature encoding module (i.e., feature maps with scales [96, 56, 56], [192, 28, 28], and [384, 14, 14]) to the same spatial dimensions as the fourth scale feature [768, 7, 7] through a pooling operation, while maintaining the channel dimension. Subsequently, the four aligned scale features are concatenated along the channel dimension to form a scale-aligned feature map with a size of [1440, 7, 7].

[0083] S22 constructs a local feature extraction module, including multiple parallel branches and a local enhancement submodule; uses the local feature extraction module to process the scale-aligned feature map to obtain a local feature map;

[0084] Based on the Inception network structure, a local feature extraction submodule is constructed, such as Figure 4 As shown in the figure, the local structural features of the original image at different spatial scales are fully modeled and extracted. Each branch uses different types of convolution to process the scale-aligned feature maps, splices the feature maps output by each branch in the channel dimension, and optimizes the spliced ​​feature maps through the local enhancement submodule to obtain the local feature map F local ;

[0085] In this embodiment, five parallel branches are used to extract local features from the scale-aligned feature map, specifically:

[0086] The first branch includes 3×3 depth-wise separable convolution and 1×1 convolution; 3×3 depth-wise separable convolution is used to reduce the number of parameters and extract edge texture features, and 1×1 convolution is used for channel mixing;

[0087] The second branch consists of 3×3 convolution, SE attention layer and 1×1 convolution; 3×3 standard convolution extracts spatial information, SE attention layer captures inter-channel dependencies, and 1×1 convolution unifies the output dimension;

[0088] The third branch includes 1×3 convolution, 3×1 convolution, coordinate convolution CoordConv and 1×1 convolution; asymmetric convolution with kernel sizes of 1×3 and 3×1 is used to extract horizontal and vertical features, coordinate convolution CoordConv is used to add explicit position information, and then 1×1 convolution is used for fusion;

[0089] The fourth branch includes 3×3 maximum pooling, 3×3 convolution, and 1×1 convolution. 3×3 maximum pooling is used to extract local context areas, followed by 3×3 convolution and 1×1 convolution to achieve feature compression and enhancement.

[0090] The fifth branch includes a 3×3 dilated convolution with a dilation rate of 2, a ReLU activation function, and a 1×1 convolution. A 3×3 dilated convolution with a dilation rate of 2 is used to expand the receptive field to capture larger-scale semantic structures, and then a ReLU activation function and a 1×1 convolution are used for processing.

[0091] The five branches described above extract local features from scale-aligned feature maps of size [1440, 7, 7]. The feature maps output by the five branches are concatenated and fused in the channel dimension. A 1×1 convolution is then used to uniformly remap the channel dimension, maintaining the overall output size of [1440, 7, 7]. To further improve the ability to respond to local salient regions in the image, a local enhancer module is added to the end of the five branches. This module generates an attention mask through 3×3 convolution. After softmax normalization, it is weightedly fused with the main branch features to strengthen the representation of key regional features. The output dimension after processing by this module remains [1440, 7, 7] and serves as the input for subsequent global path modeling and cross-granularity fusion.

[0092] S23 constructs a global feature extraction module, including a channel attention submodule, a context feature extraction submodule, and residual enhancement; uses the global feature extraction module to process the scale-aligned feature map to obtain a global feature map;

[0093] Global feature extraction module such as Figure 5 As shown in the figure, it is used to extract the global discriminant features of the original image and combine the channel attention and residual mechanisms to achieve efficient global information modeling. Among them, the channel attention submodule modulates the weights of the scale-aligned feature map in the channel dimension, thereby highlighting the response to the key feature channel; the context feature extraction submodule further models the global semantic information of the channel-modulated feature map to capture the long-distance dependent features in the image; through residual enhancement, the initial input feature map is fused with the feature map after context modeling to enhance the model's ability to maintain the original feature structure and obtain a global feature map;

[0094] In the channel attention submodule, the scale-aligned feature map is globally averaged pooled to obtain a global description vector of size [1440, 1, 1] for each channel. The channel attention weight is constructed through the fully connected mapping, and the channel attention weight is multiplied element-by-element with the scale-aligned feature map of the input channel attention submodule according to the channel dimension to generate a weighted feature map X attn ;

[0095] The context feature extraction submodule can capture global features and contextual relationships, and optimize the global feature map through further feature transformation, specifically: transforming the weighted feature map X attn The input context feature extraction submodule is flattened into a one-dimensional vector, and the hidden dimension is compressed and restored through the multi-layer perceptron. For example, the hidden layer dimension D = 1024 is restored to a three-dimensional feature map shape of size [1440, 7, 7] to obtain the context enhanced feature map X gce ;

[0096] The weighted feature map X is enhanced by residual attnand context-enhanced feature map X gce Perform element-by-element residual connection to retain local accuracy while strengthening global structure perception, and obtain a global feature map F with a size of [1440,7,7] global ;

[0097] S24 builds a cross-branch fusion module, including a channel attention submodule, a feature recalibration submodule, and a residual gated fusion submodule; local features and global features are input into the cross-branch fusion module for adaptive fusion to obtain adaptively fused features;

[0098] Construct cross-branch fusion modules, such as Figure 6 As shown, it includes a channel attention submodule, a feature recalibration submodule, and a residual gated fusion submodule, where:

[0099] The channel attention submodule is based on the global feature map F global Generate channel attention weight α local , using channel attention weight α local For the local feature map F local Perform element-by-element multiplication to obtain the enhanced local feature map F′ local , as shown in the following formula:

[0100] F′ local =α local ⊙F local

[0101] α local =Sigmoid(MLP(GAP(F global )))

[0102] Among them, α local is the channel attention weight, Sigmoid(·) is the Sigmoid activation function, MLP(·) is the multi-layer perceptron, GAP(·) is the global average pooling operation, and ⊙ is the element-wise multiplication operation;

[0103] The feature recalibration submodule sequentially recalibrates the enhanced local feature map F′ local and the global feature map F global Processing, specifically:

[0104] Use 1×1 convolution to enhance the local feature map F′ local and the global feature map F global Adjust the number of channels, use the splicing operation to fuse the adjusted local feature map and the global feature map in the channel dimension, use 3×3 convolution to extract spatial features from the fused feature map, and obtain the recalibrated feature map F through the BN layer and ReLU activation function. concat ;

[0105] The residual gated fusion submodule balances the contribution of the recalibrated features and the enhanced local features through the gating mechanism to obtain the adaptive fused feature F fused , as shown in the following formula:

[0106] F fused =G⊙F concat +(1-G)⊙F′ local

[0107] G=Sigmoid(Conv 1×1 (F concat ))

[0108] Among them, G is the weight of gate control, Conv 1×1 (·) is a 1×1 convolution;

[0109] The cross-branch fusion module enables information interaction and fusion between local and global features, improving the model's ability to model multi-granularity structural information. The scales of the local and global feature maps input to the cross-branch fusion module are both [1440, 7, 7]. First, global average pooling is performed on the global feature map to extract the response vector for each channel. This is then mapped through two layers of fully connected MLPs to generate attention weights along the channel dimension. Next, a sigmoid activation function is used to form a modulation factor, which is used to adjust the response strength of the local feature map channel by channel, thereby obtaining a weighted local feature representation. Subsequently, the adjusted local and global feature maps are processed channel-wise through 1×1 convolutions and then concatenated along the channel dimension. The concatenated result undergoes 3×3 convolutions, batch normalization, and ReLU activation to achieve feature recalibration and fusion. To further enhance information guidance, the module also introduces a gated residual mechanism: the fused feature map undergoes a 1×1 convolution and a sigmoid filter to generate a gated weight map, which serves as an information selection factor, thereby achieving a weighted integration of local enhancement and global guidance. Finally, the fusion output feature size is maintained as [1440, 7, 7], providing a unified fusion expression for the subsequent multi-branch deep interaction module.

[0110] The cross-branch fusion module, through a three-stage process involving attention enhancement, feature fusion, and gated balancing, effectively achieves a complementary fusion of local details and global context, providing a more discriminative feature representation for classification tasks. In particular, this module effectively coordinates the complementary relationship between local detail features (such as hair texture) and global structural features (such as body shape), significantly improving the completeness and discriminability of feature representation and providing an optimized feature representation for the final classification decision.

[0111] S25: Construct a multi-branch deep interaction module, including multiple parallel paths, a concatenation layer, and a 1×1 convolution layer. Input the adaptively fused features into the multi-branch deep interaction module, use multiple parallel paths to extract multi-scale features under different receptive fields, concatenate the output features of each path in the channel dimension, and then perform channel compression and feature fusion through 1×1 convolution to obtain the feature map output by the multi-branch deep interaction module.

[0112] The multi-branch deep interaction module contains three parallel paths, such as Figure 7 As shown, specifically:

[0113] The first path consists of two serial 3×3 convolutions;

[0114] The second path includes a 5×5 convolution;

[0115] The third path includes a 3×3 max pooling and a 1×1 convolution;

[0116] The multi-branch deep interaction module is used to simulate feature response mechanisms at different semantic levels and enhance the model's understanding of key discriminative regions in the original image. Taking the fused feature map of size [1440, 7, 7] output by the cross-branch fusion module as input, the module extracts spatial information at different scales through three parallel feature paths. The first branch employs two consecutive layers of 3×3 standard convolutions to model local continuous semantics; the second branch uses a single layer of 5×5 convolution to expand the receptive field and extract mid-range spatial dependencies; the third branch consists of a 3×3 max pooling layer followed by a 1×1 convolution layer to compress the space and enhance global context. After unifying the number of channels, the features output by each branch are concatenated in the channel dimension and integrated into a consistent representation through a 1×1 convolution. The final output feature dimension remains [1440, 7, 7], providing fused multi-level information support for the subsequent classification module.

[0117] S26, uses the MLP network to build a classification module, including a global average pooling layer, two fully connected layers, an intermediate batch normalization layer and a ReLU activation layer; the feature map output by the multi-branch deep interaction module is input into the classification module and the prediction result is output;

[0118] The feature map output by the multi-branch deep interaction module is input into the classification module, and the channel feature vector is obtained through global average pooling. The channel feature vector is reduced to the intermediate feature dimension through the first fully connected layer, and the predicted classification result is output in sequence through the intermediate batch normalization layer, the ReLU activation layer, and the second fully connected layer.

[0119] In this embodiment, a classification module based on an MLP network structure is used to vectorize high-dimensional feature maps and perform final classification prediction. First, a global average pooling operation is performed on the fused feature map with a scale of [1440, 7, 7] to compress the spatial dimension into a single feature vector with a scale of

[1440] . Dimension mapping and nonlinear activation are performed through two fully connected layers. The scale of the mapping process changes from

[1440] to

[256] to

[128] . Finally, a linear classifier with an output dimension of 2 is connected to obtain a prediction result with a scale of [2], which represents the probability distribution of the current image category.

[0120] S3: Use the training set to conduct supervised training on the multi-scale image recognition model. After the training is completed, the obtained model is verified using the validation set, and the model with the best performance on the validation set is selected as the final multi-scale image recognition model.

[0121] Construct an auxiliary classification module with the same structure as the classification module, apply the auxiliary classification module to the output feature map of the local feature extraction module, and use the auxiliary classification module to provide additional supervision. Both the auxiliary classification module and the classification module use the standard cross entropy loss function, as shown in the following formula:

[0122] L(y,y m )=-(ylog(y m )+(1-y)log(1-y m ))

[0123] Among them, y is the true label, y m Classification results predicted by the multi-scale image recognition model;

[0124] The multi-scale image recognition model is trained using the training set. A fully supervised learning method is used to construct the joint loss function L of the multi-scale image recognition model based on the loss function of the classification module and the loss function of the auxiliary classification module. total , as shown in the following formula:

[0125] L total =βL main +(1-β)L aux

[0126] Among them, β is the weight coefficient, L main is the loss function of the classification module, L aux is the loss function of the auxiliary classification module;

[0127] A class-weighted loss function is used to dynamically assign weights to different classes, giving higher weights to classes that appear fewer than a set threshold during training to avoid class bias. The class-weighted loss function is shown in the following formula:

[0128]

[0129] Among them, w c is the weight of category c, y c is the true label, p c is the probability of predicting category c;

[0130] End-to-end training is completed by combining the back-propagation mechanism. After the training is completed, the obtained model is verified using the validation set, and the model with the best performance on the validation set is selected as the final multi-scale image recognition model;

[0131] In this embodiment, an auxiliary classification module is introduced after the local feature extraction module to provide early discriminative supervision of local detail features, guiding the network to focus more on key areas during the intermediate stages, thereby improving the discriminative ability of the overall feature representation. The auxiliary classification module provides additional supervisory signals for the multi-scale image recognition model, helping the network optimize feature learning strategies during training and ultimately improving overall classification performance.

[0132] In this embodiment, the weight in the loss function is adjusted to balance the contribution of the main classification module and the auxiliary classification module to the training process, and the weight coefficient β is set to 0.7;

[0133] S4, after uniform preprocessing of the image to be classified, input it into the trained multi-scale image recognition model and output the recognition result.

[0134] To demonstrate the effectiveness of the multi-scale image recognition model provided in this embodiment, a cat-dog classification dataset was constructed based on the cat-dog classification task. A comparative experiment was conducted on the constructed cat-dog classification dataset using the multi-scale image recognition model provided in this embodiment and a common deep learning model. The results of the comparative experiment are shown in Table 1, using accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the receiver operating characteristic (ROC) curve (AUC) as evaluation indicators.

[0135] Table 1 Comparative experimental results

[0136]

[0137]

[0138] As can be seen from the table, compared with common visual classification task models, the multi-scale image recognition model provided by this embodiment has obvious advantages in ACC, SEN, SPE, and AUC indicators.

[0139] By using the technical means provided by this invention, relevant technicians can build a complete software system or deploy it to terminal recognition devices, thereby achieving efficient and robust classification and recognition of images. Therefore, all hardware platforms, software systems, and application solutions based on this method involved in this embodiment are within the scope of protection of this invention.

[0140] It should be understood that the above specific embodiments are intended only to illustrate the core principles and implementation paths of the present invention and to facilitate understanding of the design principles of the proposed image classification system. Those skilled in the art may make equivalent adjustments and extensions to the details of the above method without departing from the spirit and substance of the present invention, and such modifications should be considered within the technical scope of the present invention.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An image recognition method based on multi-scale feature fusion, characterized by: The following steps are involved: S1, obtain original image data, perform preprocessing and enhancement on the obtained original image data, and construct a classification data set; Divide the classification dataset into training, validation, and test sets in proportion; S2, builds a multi-scale image recognition model, including: feature encoding module, local feature extraction module, global feature extraction module, cross-branch fusion module, multi-branch deep interaction module and classification module; For the original image input into the multi-scale image recognition model, the feature encoding module is used to extract the multi-scale feature map that is refined layer by layer, and the multi-scale feature map is unified to the smallest spatial size among all scales. The scale-aligned feature map is obtained by channel splicing; the scale-aligned feature map is transmitted to the local feature extraction module and the global feature extraction module respectively to extract the local feature map and the global feature map respectively; the local feature map and the global feature map are passed to the cross-branch fusion module for fusion to generate a fused feature map, which is transmitted to the multi-branch deep interaction module to further optimize the fused feature map. The optimized feature map is input into the classification module to obtain the classification result of the original image; S3, use the training set to supervise the multi-scale image recognition model. After the training is completed, the obtained model is verified using the validation set, and the model with the best performance on the validation set is selected as the trained multi-scale image recognition model; S4, after uniform preprocessing of the image to be classified, input it into the trained multi-scale image recognition model and output the recognition result.

2. The image recognition method based on multi-scale feature fusion according to claim 1, characterized in that: S1, the preprocessing includes: unifying the size of the original image obtained, standardizing, center cropping and boundary clearing the original image after the unified size, to obtain a preprocessed original image; The enhancement processing includes: enhancing the pre-processed original image by using random rotation, flipping and brightness perturbation methods.

3. The image recognition method based on multi-scale feature fusion according to claim 2, characterized in that: The S2 includes: S21, use the Swin-Transformer network to build a feature encoding module, use the feature encoding module to perform multi-scale feature extraction, scale alignment and feature splicing on the input original image to obtain a scale-aligned feature map; S22, constructing a local feature extraction module, including multiple parallel branches and a local enhancement submodule; using the local feature extraction module to process the scale-aligned feature map to obtain a local feature map; S23, constructing a global feature extraction module, including a channel attention submodule, a context feature extraction submodule, and residual enhancement; using the global feature extraction module to process the scale-aligned feature map to obtain a global feature map; S24, constructing a cross-branch fusion module, including a channel attention submodule, a feature recalibration submodule, and a residual gated fusion submodule; inputting local features and global features into the cross-branch fusion module for adaptive fusion to obtain adaptively fused features; S25: Construct a multi-branch deep interaction module, including multiple parallel paths, a concatenation layer, and a 1×1 convolution layer. Input the adaptively fused features into the multi-branch deep interaction module, use multiple parallel paths to extract multi-scale features under different receptive fields, concatenate the output features of each path in the channel dimension, and then perform channel compression and feature fusion through 1×1 convolution to obtain the feature map output by the multi-branch deep interaction module. S26, use the MLP network to build a classification module, including a global average pooling layer, two fully connected layers, an intermediate batch normalization layer and a ReLU activation layer; input the features output by the multi-branch deep interaction module into the classification module and output the prediction results.

4. The image recognition method based on multi-scale feature fusion according to claim 3, characterized in that: The S21 includes: The feature encoding module is constructed using the Swin-Transformer network. The Swin-Transformer network includes multiple basic encoding units and scale alignment and feature splicing units. Each basic encoding unit includes a patch merging submodule, a local window self-attention submodule, and a sliding window self-attention submodule. The scale alignment and feature splicing unit includes a pooling downsampling layer and a splicing layer. Use the feature encoding module to perform multi-scale feature extraction, scale alignment, and feature splicing on the input original image to obtain a scale-aligned feature map. The specific method is as follows: In the Patch Merging submodule, the original image input to the Patch Merging submodule is divided into 4×4 regions. Four spatially adjacent 2×2 regions are divided into a patch. The feature vectors of each region in the patch are extracted. The feature vectors of these four regions are concatenated in the channel dimension and then linearly transformed to obtain a feature map with half the width and height and double the number of channels. In the local window self-attention submodule, for a feature map X∈R with C channels and H×W size H×W×C , divide the feature map X into N local windows of size M×M, perform single-head self-attention calculation on each local window, and the single-head self-attention head of the nth local window is n ; Concatenate and linearly project the single-head attentions of N local windows to obtain a multi-head self-attention feature map; In the sliding window self-attention submodule, the sliding window size is set to M×M, and the initial position of the sliding window is offset by a fixed step size relative to the sliding window position of the sliding window self-attention submodule in the previous basic coding unit. The calculation method of the sliding window self-attention feature map is the same as the attention method within the local window, only the input window area is different. The feature map X with C channels and size H×W passes through the local window self-attention submodule and the sliding window self-attention submodule in sequence to obtain the feature map output by the basic coding unit. For the multiple basic coding units of the Swin-Transformer network, which output multiple feature maps of different scales, the feature map with the smallest scale is used as the baseline. The pooling downsampling layer is used to perform pooling downsampling operations on the feature maps of the remaining scales to keep the scales of the feature maps output by the multiple basic coding units consistent. The splicing layer is then used to splice the feature maps with the same scale in the channel dimension to obtain the scale-aligned feature map F. aligned .

5. The image recognition method based on multi-scale feature fusion according to claim 4, characterized in that: The S22 includes: The local feature extraction module contains five parallel branches, specifically: The first branch includes 3×3 depth-wise separable convolution and 1×1 convolution; The second branch includes 3×3 convolution, SE attention layer and 1×1 convolution; The third branch includes 1×3 convolution, 3×1 convolution, coordinate convolution CoordConv and 1×1 convolution; The fourth branch includes 3×3 maximum pooling, 3×3 convolution and 1×1 convolution; The fifth branch includes a 3×3 dilated convolution with a dilation rate of 2, a ReLU activation function, and a 1×1 convolution; The local enhancement submodule includes a 3×3 convolution, a Softmax normalization layer, and a weighted fusion layer; The five branches of the local feature extraction module use different types of convolution to align the scale feature map F aligned Processing is performed, the feature maps output by each branch are spliced ​​in the channel dimension, and the spliced ​​feature maps are optimized through the local enhancement submodule to obtain the local feature map F local .

6. The image recognition method based on multi-scale feature fusion according to claim 5, characterized in that: The S23 includes: In the channel attention submodule, the scale-aligned feature map F aligned Perform global average pooling to obtain the global description vector of each channel, and construct the channel attention weight through the fully connected mapping layer. Multiply the channel attention weight with the feature map aligned with the scale of the input channel attention submodule by element according to the channel dimension to generate the weighted feature map X attn ; The weighted feature map X attn The input context feature extraction submodule is flattened into a one-dimensional vector, and the hidden dimension is compressed and restored through the multi-layer perceptron to obtain the context enhanced feature map X gce ; The weighted feature map X is enhanced by residual attn and context-enhanced feature map X gce Perform element-by-element residual connection to obtain the global feature map F global .

7. The image recognition method based on multi-scale feature fusion according to claim 6, characterized in that: The S24 includes: The channel attention submodule is based on the global feature map F global Generate channel attention weight α local , using channel attention weight α local For the local feature map F local Perform element-by-element multiplication to obtain the enhanced local feature map F′ local ; The feature recalibration submodule sequentially recalibrates the enhanced local feature map F′ local and the global feature map F global Processing, specifically: Use 1×1 convolution to enhance the local feature map F′ local and the global feature map F global Adjust the number of channels, use the splicing operation to fuse the adjusted local feature map and the global feature map in the channel dimension, use 3×3 convolution to extract spatial features from the fused feature map, and obtain the recalibrated feature map F through the BN layer and ReLU activation function. concat ; The residual gated fusion submodule balances the contribution of the recalibrated features and the enhanced local features through the gating mechanism to obtain the adaptively fused feature map F fused .

8. The image recognition method based on multi-scale feature fusion according to claim 7, characterized in that: The S25 includes: The multi-branch deep interaction module contains three parallel paths, specifically: The first path consists of two serial 3×3 convolutions; The second path includes a 5×5 convolution; The third path consists of a 3×3 max pooling and a 1×1 convolution.

9. The image recognition method based on multi-scale feature fusion according to claim 8, characterized in that: The S26 includes: The feature map output by the multi-branch deep interaction module is input into the classification module, and the channel feature vector is obtained by global average pooling; the channel feature vector is reduced to the intermediate feature dimension through the first fully connected layer, and the predicted classification result is output in sequence through the intermediate batch normalization layer, the ReLU activation layer, and the second fully connected layer.

10. The image recognition method based on multi-scale feature fusion according to claim 9, characterized in that: The S3 includes: Construct an auxiliary classification module with the same structure as the classification module, apply the auxiliary classification module to the output feature map of the local feature extraction module, and use the auxiliary classification module to provide additional supervision. Both the auxiliary classification module and the classification module use the standard cross entropy loss function. The multi-scale image recognition model is trained using the training set. A fully supervised learning method is adopted to construct a joint loss function of the multi-scale image recognition model based on the loss function of the classification module and the loss function of the auxiliary classification module. A weighted loss function strategy is adopted to dynamically assign weights to different categories, giving higher weights to categories that appear less than the set threshold during training. The end-to-end training is completed in combination with the back-propagation mechanism. After the training is completed, the obtained model is verified using the validation set, and the model with the best performance on the validation set is selected as the final multi-scale image recognition model.

Images