Image classification identification method based on adaptive multi-scale feature fusion network

Abstract

An image classification and identification method based on a self-adaptive multi-scale feature fusion network is characterized by comprising the following steps: step 1, an image acquisition module acquires image data a and transmits the image data a to the self-adaptive multi-scale feature fusion network; 2, performing feature extraction operation on the image data a by a feature extraction module in the adaptive multi-scale feature fusion network to obtain feature data b, and transmitting the feature data b to a feature fusion module; 3, performing feature fusion operation on the feature data b by the feature fusion module to obtain fusion data c, and transmitting the fusion data c to a classification module; and step 4, the classification module classifies and identifies the fusion data c to obtain a classification result. The method has the advantages that redundant information and noise are effectively eliminated, the representation capability of image features is enhanced, multi-level and multi-scale features of the CT image are effectively extracted, semantic correlation between features with different resolutions is enhanced, and the accuracy and reliability of a prediction result are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image classification and recognition method based on an adaptive multi-scale feature fusion network. Background Art

[0002] Image classification is to distinguish different categories of images based on their semantic information. It is the core of computer vision and the basis of other high-level visual tasks such as object detection, image segmentation, object tracking, behavior analysis, face recognition, etc. Image classification has a wide range of applications in many fields, such as face recognition and intelligent video analysis in the security field, traffic scene recognition in the transportation field, content-based image retrieval and automatic album classification in the Internet field, and image recognition in the medical field.

[0003] In the early 2000s, although neural networks began to show signs of recovery, they were still very difficult to train and optimize due to the size of the data set and the development of hardware. Both the MNIST and CIFAR data sets have only 60,000 images, which may be enough for simple tasks such as 10 classifications, but it is still far from enough if you want to implement more complex image classification tasks in the industry.

[0004] Later, after years of research by Fei-Fei Li and others, the ImageNet dataset was released in 2009, and the ImageNet Large Scale Visual Recognition Challenge, or ILSVRC, has been held annually since 2010. The ImageNet dataset contains more than 14 million images, covering more than 20,000 categories. The 1,000-category benchmark is often used in the comparison of paper methods.

[0005] In the early years after ImageNet was released, classification methods represented by SVM and Boost still dominated until the emergence of AlexNet in 2012. AlexNet was the first truly deep network. Compared with LeNet5's 5 layers, it had 3 more layers, and the number of network parameters also increased greatly, with the input increasing from 28 to 224. At the same time, the advent of GPUs also ushered in an era of GPU-dominated training for deep learning.

[0006] After that, various excellent networks continued to emerge, such as VGG, ResNet and other excellent networks, and later ConvNext and others continued to refresh the ImageNet record. The accuracy of image classification has rapidly developed from less than 50% to a high accuracy of nearly 90%.

[0007] Disadvantages of existing technologies: Current image classification and recognition methods are not capable of extracting multi-scale features and are unable to accurately obtain multi-level and multi-scale image features, which in turn leads to insufficient classification accuracy in complex image classification and recognition tasks. Summary of the invention

[0008] The present invention provides an image classification and recognition method based on an adaptive multi-scale feature fusion network, which improves the accuracy and reliability of image classification and recognition.

[0009] To achieve the above object, the present invention provides an image classification and recognition method based on an adaptive multi-scale feature fusion network, the key of which is to include the following steps:

[0010] Step 1: The image acquisition module acquires image data a and passes it to the adaptive multi-scale feature fusion network;

[0011] Step 2: The feature extraction module in the adaptive multi-scale feature fusion network performs a feature extraction operation on the image data a to obtain feature data b, and passes it to the feature fusion module;

[0012] Step 3: The feature fusion module performs a feature fusion operation on the feature data b to obtain fused data c, and passes it to the classification module;

[0013] Step 4: The classification module classifies and identifies the fused data c to obtain a classification result.

[0014] Through the above design, the feature extraction module performs four-scale transformation on the image data based on the main features of various types of image data through four adaptive multi-branch attention modules AMA to obtain feature maps of four different scales to capture the different granularity level features of the image data from coarse to fine. As the dimension of the feature map gradually decreases, the extracted feature information gradually becomes richer from low to high, thereby capturing the multi-level and multi-scale information of the image. The acquisition of this multi-level and multi-scale feature enables the adaptive multi-scale feature fusion network to better understand the complex details and overall background in the image, laying a solid foundation for the subsequent feature fusion and classification steps.

[0015] The feature fusion module achieves fast bidirectional fusion of high-level and low-level information through dynamic weights. For irregular image data, multi-scale feature fusion can integrate such complex features, thereby improving model performance and classification accuracy.

[0016] Preferably, in step 2, the feature extraction module is provided with four adaptive multi-branch attention modules AMA, the four adaptive multi-branch attention modules AMA have the same structure and are connected head to tail in sequence;

[0017] The feature extraction module extracts features from the image data a, and the specific steps are as follows:

[0018] Step 21: The first AMA1 in the feature extraction module acquires the image data a, performs feature extraction on the image data a, obtains the first feature data a1, and passes it to the second AMA2 and the first 1×1 convolutional layer;

[0019] Step 22: the second AMA2 performs feature extraction on the first feature data a1 to obtain second feature data a2, and passes it to the third AMA3 and the second 1×1 convolutional layer;

[0020] Step 23: the third AMA3 performs feature extraction on the second feature data a2 to obtain third feature data a3, and passes it to the fourth AMA4 and the third 1×1 convolutional layer;

[0021] Step 24: the fourth AMA4 performs feature extraction on the third feature data a3 to obtain fourth feature data a4, and passes it to the fourth 1×1 convolutional layer;

[0022] Step 25: The first 1×1 convolution layer performs a one-dimensional convolution operation on the first feature data a1 to obtain first convolution data b1, and passes it to the feature fusion module;

[0023] The second 1×1 convolution layer performs a one-dimensional convolution operation on the second feature data a2 to obtain second convolution data b2, and passes it to the feature fusion module;

[0024] The third 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain third convolution data b3, and passes it to the feature fusion module;

[0025] The fourth 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain fourth convolution data b4, and passes it to the feature fusion module.

[0026] Preferably, in step 21, the first AMA1 extracts features from the image data a, and the specific steps are as follows:

[0027] Step 211: the depth separable layer in the first AMA1 obtains the image data a, performs depth separation on the image data a, obtains depth separation data x1, and passes it to the first batch of normalization layers;

[0028] Step 212: the first batch of normalization layers performs batch normalization processing on the depth separation data x1 to obtain a first batch of normalized data x2, and passes it to the convolutional attention mechanism CBAM, the global context attention mechanism GC and the fifth 1×1 convolutional layer;

[0029] Step 213a: the convolutional attention mechanism CBAM performs convolutional attention adjustment on the first batch of normalized data x2 to obtain convolutional attention data x3, and passes it to the second batch of normalized layers;

[0030] Step 213b: the global context attention mechanism GC performs global context attention adjustment on the first batch of normalized data x2 to obtain global attention data x4, and passes it to the third batch of normalized layers;

[0031] Step 213c: the fifth 1×1 convolution layer performs a one-dimensional convolution operation on the first batch of normalized data x2 to obtain fifth convolution data x5, and passes it to the fourth batch normalization layer;

[0032] Step 214a: The second batch normalization layer performs batch normalization processing on the convolution attention data x3, and then activates it through the linear rectification function ReLu to obtain the first activation data y1, and passes it to the fusion module;

[0033] Step 214b: The third batch normalization layer performs batch normalization processing on the global attention data x4, and then activates it through a linear rectification function ReLu to obtain second activation data y2, and passes it to the fusion module;

[0034] Step 214c: the fourth batch normalization layer performs batch normalization processing on the fifth convolution data x5, and then activates it through the layer normalization function ReLu to obtain the third activation data y3, and passes it to the fusion module;

[0035] Step 215: The fusion module fuses the first activation data y1, the second activation data y2 and the third activation data y3 to obtain first feature data a1.

[0036] Through the above design, in each adaptive multi-branch attention module AMA, a depth-separable layer with a kernel size of 3×3 and a batch normalization layer are first used to generate a feature map with a dimension half of the input feature map. This dimensionality reduction strategy aims to achieve multi-scale feature extraction, thereby providing richer information for image classification and recognition tasks. Subsequently, the generated feature maps are fed into three different branches respectively.

[0037] In order to extract features from both channel and spatial dimensions, the convolutional attention mechanism CBAM is introduced in the first branch; the global contextual attention mechanism GC is used in the second branch to focus the network on the overall structure and contextual semantic information of the image; the 1×1 convolution is introduced in the third branch. The use of 1×1 convolution enhances the network's ability to represent complex features by introducing additional nonlinear transformations. Finally, the feature maps obtained from the three branches are fused. The fusion weights are dynamically trainable parameters that facilitate adaptive adjustment and further improve the effectiveness of feature fusion. After processing by four AMAs, the network obtains high-level abstract features and retains relatively original information from lower layers.

[0038] Preferably, in step 213a, the convolutional attention mechanism CBAM is provided with a channel attention mechanism and a spatial attention mechanism, and the convolutional attention mechanism CBAM performs a convolutional attention adjustment process as follows:

[0039] The channel attention mechanism multiplies the first batch of normalized data x2 and the channel attention map Mc to obtain the channel attention data M; the spatial attention mechanism multiplies the channel attention data and the spatial attention map Ms to obtain the convolution attention data x3.

[0040] The basic concept of the convolutional attention mechanism (CBAM) is to combine channel attention and spatial attention mechanisms to ensure that the network can focus on key features in both channel and spatial dimensions. This design aims to eliminate redundant information and noise and enhance the representation ability of image features.

[0041] Preferably, the channel attention map Mc is obtained by the following steps:

[0042] Step A1a: the first maximum pooling layer in the channel attention mechanism performs a maximum pooling operation on the first batch of normalized data x2 to obtain the first maximum pooling data c1, and passes it to the multi-layer perceptron;

[0043] Step A1b: the first average pooling layer in the channel attention mechanism performs an average pooling operation on the first batch of normalized data x2 to obtain the first average pooling data c2, and passes it to the multi-layer perceptron;

[0044] Step A2: the multi-layer perceptron performs multi-layer perception on the first maximum pooling data c1 and the first average pooling data c2 to obtain perception data c3, and passes it to the second maximum pooling layer and the second average pooling layer;

[0045] Step A3a: the second maximum pooling layer performs a maximum pooling operation on the perception data c3 to obtain second maximum pooling data c4, and passes it to the first element-by-element addition unit;

[0046] Step A3b: the second average pooling layer performs an average pooling operation on the perception data c3 to obtain second average pooling data c5, and passes it to the first element-by-element addition unit;

[0047] Step A4: The first element-by-element addition unit adds the second maximum pooling data c4 and the second average pooling data c5 element-by-element, and then activates them through a sigmoid function to obtain a channel attention map Mc.

[0048] The calculation equation of the channel attention map Mc is as follows:

[0049]

[0050] Among them, σ represents the sigmoid function, W0 and W1 represent the weights of the multi-layer perceptron, MLP represents the multi-layer perceptron, AvgPool represents average pooling, and MaxPool represents maximum pooling; F1 represents the input feature, that is, the first batch of normalized data x2; and Indicates channel information.

[0051] Preferably, the spatial attention map Ms is obtained by the following steps:

[0052] Step B1: the third maximum pooling layer in the spatial attention mechanism performs a maximum pooling operation on the channel attention map Mc to obtain the third maximum pooling data s1, and passes it to the third average pooling layer;

[0053] Step B2: the third average pooling layer performs an average pooling operation on the channel attention map Mc to obtain the third average pooling data s2, and passes it to the convolution layer;

[0054] Step B3: the convolution layer performs a convolution operation on the third average pooling data s2 to obtain convolution data s3, and passes it to the fourth maximum pooling layer;

[0055] Step B4: the fourth maximum pooling layer performs a maximum pooling operation on the convolution data s3 to obtain fourth maximum pooling data s4, and passes it to the fourth average pooling layer;

[0056] Step B5: The fourth average pooling layer performs an average pooling operation on the fourth maximum pooling data s4, and then activates it through a sigmoid function to obtain a spatial attention map Ms.

[0057] The calculation equation of the spatial attention map Ms is as follows:

[0058]

[0059] Among them, f7×7 represents a convolution operation with a filter size of 7×7, and Represents spatial information.

[0060] Preferably, in step 213b, the global context attention mechanism GC is provided with a context modeling module and a conversion module, and the global context attention mechanism GC performs global context attention adjustment, and the specific steps are as follows:

[0061] Step C1: the sixth 1×1 convolutional layer in the context modeling module performs a convolution operation on the first batch of normalized data x2, and then activates it through a sigmoid function to obtain fourth activation data d1, and then multiplies the fourth activation data d1 and the first batch of normalized data x2 element by element to obtain modeling data d2, and passes it to the conversion module;

[0062] Step C2: the seventh 1×1 convolutional layer in the conversion module performs a convolution operation on the modeling data d2, and then activates the modeling data d2 through a layer normalization function ReLu, and then performs a convolution operation on the activated data through an eighth 1×1 convolutional layer to obtain conversion data d3, and pass it to the second element-by-element addition unit;

[0063] Step C3: The second element-by-element addition unit adds the first batch of normalized data x2 and the conversion data d3 element-by-element to obtain global attention data x4.

[0064] The global contextual attention mechanism GC operates from a global perspective, aiming to enhance the neural network's abstract understanding of image features, alleviate the loss of global information, and focus on capturing the global structure and contextual semantic features of the image.

[0065] In the context modeling module, the original feature Q, i.e. the first batch of normalized data x2, is convolved and then softmaxed to generate a global probability distribution K, i.e. the fourth activation data d1, which describes the importance of each position in the image to the GC. It effectively presents the context weight of each position, which helps the network better understand the key positions in the image to determine the image category. The obtained probability weight is then multiplied element-wise by the original feature map. This operation generates a feature with context weights, introducing the influence of GC at each position. Through this step, the model emphasizes the relationship between each position and the global information, thereby better capturing the semantic connection between different positions within the image.

[0066] In the conversion module, convolution and layer normalization operations are applied to the context-weighted features to produce a feature map V containing GC information, namely the conversion data d3. This normalization improves the convergence speed and generalization ability of the model. Finally, the global information in V is combined with the original feature Q to obtain the output feature containing GC influence, namely the global attention data x4. This helps the network better understand the effective features of different types of image data.

[0067] The global context attention mechanism GC injects richer global semantic information into the neural network by generating global probability distribution, introducing GC attention and feature normalization, effectively improving the performance of the model. The specific modeling process of the global context attention mechanism GC is as follows:

[0068] K=s(resize(f 1×1 (Q)))

[0069]

[0070] M o =Q+V

[0071] Among them, Q represents the input feature map of the GC module, that is, the first batch of normalized data x2; M o represents the output feature map of the GC module, i.e., the global attention data x4; resize(·) represents the shape transformation operation, s(·) represents the softmax function, g(·) represents the layer normalization and ReLU function, represents element-by-element multiplication, f 1×1 Represents a convolution operation with a filter size of 1×1.

[0072] Preferably, in step 3, the feature fusion module is provided with a first feature fusion layer, a second feature fusion layer, a third feature fusion layer and a fourth feature fusion layer, wherein the first feature fusion layer is provided with a first acquisition unit and a first fusion unit, the second feature fusion layer is provided with a second acquisition unit, a second fusion unit and a third fusion unit, the third feature fusion layer is provided with a third acquisition unit, a fourth fusion unit and a fifth fusion unit, and the fourth feature fusion layer is provided with a fourth acquisition unit and a sixth fusion unit;

[0073] The feature fusion module performs feature fusion operation, and the specific steps are as follows:

[0074] Step 31: The first acquisition unit acquires the first convolution data b1 and transmits it to the first fusion unit;

[0075] The second acquisition unit acquires the second convolution data b2 and transmits it to the second fusion unit and the third fusion unit;

[0076] The third acquisition unit acquires the third convolution data b3 and passes it to the fourth fusion unit and the fifth fusion unit;

[0077] The fourth acquisition unit acquires the fourth convolution data b4 and transmits it to the fourth fusion unit and the sixth fusion unit;

[0078] Step 32: the fourth fusion unit performs feature fusion on the third convolution data b3 and the fourth convolution data b4 to obtain fourth fusion data f4, and transmits the fourth fusion data f4 to the second fusion unit and the fifth fusion unit;

[0079] The second fusion unit performs feature fusion on the second convolution data b2 and the fourth fusion data f4 to obtain second fusion data f2, and transmits the second fusion data f2 to the first fusion unit and the third fusion unit;

[0080] Step 33: the first fusion unit performs feature fusion on the first convolution data b1 and the second fusion data f2 to obtain first fusion data f1, and passes it to the third fusion unit and the third element-by-element addition unit;

[0081] The third fusion unit performs feature fusion on the second convolution data b2, the first fusion data f1 and the second fusion data f2 to obtain third fusion data f3, and passes the third fusion data f3 to the fifth fusion unit and the third element-by-element addition unit;

[0082] The fifth fusion unit performs feature fusion on the third convolution data b3, the fourth fusion data f4 and the third fusion data f3 to obtain fifth fusion data f5, and passes the fifth fusion data f5 to the sixth fusion unit and the third element-by-element addition unit;

[0083] The sixth fusion unit performs feature fusion on the fourth convolution data b4 and the fifth fusion data f5 to obtain sixth fusion data f6, and passes the sixth fusion data f6 to the third element-by-element addition unit;

[0084] Step 34: the third element-by-element adding unit adds the first fused data f1, the third fused data f3, the fifth fused data f5 and the sixth fused data f6 element-by-element to obtain fused data c.

[0085] Through the above design, the feature fusion module adopts a weighted bidirectional feature pyramid network BiFPN to strengthen the semantic correlation between features of different resolutions, realizes top-down and bottom-up bidirectional information flow, and each fusion path is equipped with trainable parameters to automatically adjust the optimal fusion ratio. For information of different scales from the feature extraction module, this method adopts unified upsampling and downsampling operations to ensure that they obtain the same feature resolution. Regarding upsampling, this method selects a stable and smooth bilinear interpolation method, while downsampling involves maximum pooling. In addition, between the input and output nodes of the same layer, jump connections are inserted to alleviate the degradation and information loss caused by information propagation.

[0086] Preferably, in step 4, the classification module is provided with a first fully connected layer and a second fully connected layer;

[0087] The first fully connected layer performs a fully connected operation on the fused data c to obtain fully connected data and passes it to the second fully connected layer; the second fully connected layer performs a fully connected operation on the fully connected data to obtain a classification result.

[0088] Preferably, the classification result is a 1×Q vector composed of 0 and 1, and the number on each vector position in the vector is used to define an image type;

[0089] In the vector, only one vector position has a number of 1, and the image type corresponding to the vector position is the classification result.

[0090] Through the above design, classification and recognition of different types of image data are achieved, and according to the prediction results, it can be intuitively and clearly understood to which image category the predicted image specifically belongs.

[0091] In practical applications, the corresponding image types can be set according to the specific image classification scenario, and then the adaptive multi-scale feature fusion network can be trained accordingly to achieve accurate classification of the corresponding image data. For example, when animal images are used for classification training, the animal species information in the image data can be accurately identified, such as: cats, dogs, horses, sheep, fish, etc.

[0092] The beneficial effects of the present invention are as follows: in the feature extraction module, key features are focused on in the channel and spatial dimensions respectively, redundant information and noise are effectively eliminated, the representation capability of image features is enhanced, and effective extraction of multi-level and multi-scale features of CT images is achieved; in the feature fusion module, BiFPN is introduced to fuse multi-scale features, the semantic correlation between features of different resolutions is enhanced, and the accuracy and reliability of the prediction results are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a schematic diagram of the process of the present invention;

[0094] Figure 2 Schematic diagram of the adaptive multi-scale feature fusion network structure;

[0095] Figure 3 Schematic diagram of the convolutional attention mechanism CBAM structure;

[0096] Figure 4 This is a schematic diagram of the global context attention mechanism GC structure;

[0097] Figure 5 Schematic diagram of the feature fusion module structure. DETAILED DESCRIPTION

[0098] The present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. The following examples or drawings are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0099] Embodiment 1:

[0100] like Figure 1 As shown: An image classification and recognition method based on an adaptive multi-scale feature fusion network, characterized in that it includes the following steps:

[0101] Step 1: The image acquisition module acquires image data a and passes it to the adaptive multi-scale feature fusion network;

[0102] Step 2: The feature extraction module in the adaptive multi-scale feature fusion network performs a feature extraction operation on the image data a to obtain feature data b, and passes it to the feature fusion module;

[0103] Step 3: The feature fusion module performs a feature fusion operation on the feature data b to obtain fused data c, and passes it to the classification module;

[0104] Step 4: The classification module classifies and identifies the fused data c to obtain a classification result.

[0105] like Figure 2 As shown: in the step 2, the feature extraction module is provided with four adaptive multi-branch attention modules AMA, and the four adaptive multi-branch attention modules AMA have the same structure and are connected head to tail in sequence;

[0106] The feature extraction module extracts features from the image data a, and the specific steps are as follows:

[0107] Step 21: The first AMA1 in the feature extraction module acquires the image data a, performs feature extraction on the image data a, obtains the first feature data a1, and passes it to the second AMA2 and the first 1×1 convolutional layer;

[0108] Step 22: the second AMA2 performs feature extraction on the first feature data a1 to obtain second feature data a2, and passes it to the third AMA3 and the second 1×1 convolutional layer;

[0109] Step 23: the third AMA3 performs feature extraction on the second feature data a2 to obtain third feature data a3, and passes it to the fourth AMA4 and the third 1×1 convolutional layer;

[0110] Step 24: the fourth AMA4 performs feature extraction on the third feature data a3 to obtain fourth feature data a4, and passes it to the fourth 1×1 convolutional layer;

[0111] Step 25: The first 1×1 convolution layer performs a one-dimensional convolution operation on the first feature data a1 to obtain first convolution data b1, and passes it to the feature fusion module;

[0112] The second 1×1 convolution layer performs a one-dimensional convolution operation on the second feature data a2 to obtain second convolution data b2, and passes it to the feature fusion module;

[0113] The third 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain third convolution data b3, and passes it to the feature fusion module;

[0114] The fourth 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain fourth convolution data b4, and passes it to the feature fusion module.

[0115] In the step 21, the first AMA1 performs feature extraction on the image data a, and the specific steps are as follows:

[0116] Step 211: the depth separable layer in the first AMA1 obtains the image data a, performs depth separation on the image data a, obtains depth separation data x1, and passes it to the first batch of normalization layers;

[0117] Step 212: the first batch of normalization layers performs batch normalization processing on the depth separation data x1 to obtain a first batch of normalized data x2, and passes it to the convolutional attention mechanism CBAM, the global context attention mechanism GC and the fifth 1×1 convolutional layer;

[0118] Step 213a: the convolutional attention mechanism CBAM performs convolutional attention adjustment on the first batch of normalized data x2 to obtain convolutional attention data x3, and passes it to the second batch of normalized layers;

[0119] Step 213b: the global context attention mechanism GC performs global context attention adjustment on the first batch of normalized data x2 to obtain global attention data x4, and passes it to the third batch of normalized layers;

[0120] Step 213c: the fifth 1×1 convolution layer performs a one-dimensional convolution operation on the first batch of normalized data x2 to obtain fifth convolution data x5, and passes it to the fourth batch normalization layer;

[0121] Step 214a: The second batch normalization layer performs batch normalization processing on the convolution attention data x3, and then activates it through the linear rectification function ReLu to obtain the first activation data y1, and passes it to the fusion module;

[0122] Step 214b: The third batch normalization layer performs batch normalization processing on the global attention data x4, and then activates it through a linear rectification function ReLu to obtain second activation data y2, and passes it to the fusion module;

[0123] Step 214c: the fourth batch normalization layer performs batch normalization processing on the fifth convolution data x5, and then activates it through the layer normalization function ReLu to obtain the third activation data y3, and passes it to the fusion module;

[0124] Step 215: The fusion module fuses the first activation data y1, the second activation data y2 and the third activation data y3 to obtain first feature data a1.

[0125] like Figure 3 As shown: In step 213a, the convolutional attention mechanism CBAM is provided with a channel attention mechanism and a spatial attention mechanism, and the convolutional attention mechanism CBAM performs the convolutional attention adjustment process as follows:

[0126] The channel attention mechanism multiplies the first batch of normalized data x2 and the channel attention map Mc to obtain the channel attention data M; the spatial attention mechanism multiplies the channel attention data and the spatial attention map Ms to obtain the convolution attention data x3.

[0127] The channel attention map Mc is obtained by the following steps:

[0128] Step A1a: the first maximum pooling layer in the channel attention mechanism performs a maximum pooling operation on the first batch of normalized data x2 to obtain the first maximum pooling data c1, and passes it to the multi-layer perceptron;

[0129] Step A1b: the first average pooling layer in the channel attention mechanism performs an average pooling operation on the first batch of normalized data x2 to obtain the first average pooling data c2, and passes it to the multi-layer perceptron;

[0130] Step A2: the multi-layer perceptron performs multi-layer perception on the first maximum pooling data c1 and the first average pooling data c2 to obtain perception data c3, and passes it to the second maximum pooling layer and the second average pooling layer;

[0131] Step A3a: the second maximum pooling layer performs a maximum pooling operation on the perception data c3 to obtain second maximum pooling data c4, and passes it to the first element-by-element addition unit;

[0132] Step A3b: the second average pooling layer performs an average pooling operation on the perception data c3 to obtain second average pooling data c5, and passes it to the first element-by-element addition unit;

[0133] Step A4: The first element-by-element addition unit adds the second maximum pooling data c4 and the second average pooling data c5 element-by-element, and then activates them through a sigmoid function to obtain a channel attention map Mc.

[0134] The spatial attention map Ms is obtained by the following steps:

[0135] Step B1: the third maximum pooling layer in the spatial attention mechanism performs a maximum pooling operation on the channel attention map Mc to obtain the third maximum pooling data s1, and passes it to the third average pooling layer;

[0136] Step B2: the third average pooling layer performs an average pooling operation on the channel attention map Mc to obtain the third average pooling data s2, and passes it to the convolution layer;

[0137] Step B3: the convolution layer performs a convolution operation on the third average pooling data s2 to obtain convolution data s3, and passes it to the fourth maximum pooling layer;

[0138] Step B4: the fourth maximum pooling layer performs a maximum pooling operation on the convolution data s3 to obtain fourth maximum pooling data s4, and passes it to the fourth average pooling layer;

[0139] Step B5: The fourth average pooling layer performs an average pooling operation on the fourth maximum pooling data s4, and then activates it through a sigmoid function to obtain a spatial attention map Ms.

[0140] like Figure 4As shown: In step 213b, the global context attention mechanism GC is provided with a context modeling module and a conversion module, and the global context attention mechanism GC performs global context attention adjustment, and the specific steps are as follows:

[0141] Step C1: the sixth 1×1 convolutional layer in the context modeling module performs a convolution operation on the first batch of normalized data x2, and then activates it through a sigmoid function to obtain fourth activation data d1, and then multiplies the fourth activation data d1 and the first batch of normalized data x2 element by element to obtain modeling data d2, and passes it to the conversion module;

[0142] Step C2: the seventh 1×1 convolutional layer in the conversion module performs a convolution operation on the modeling data d2, and then activates the modeling data d2 through a layer normalization function ReLu, and then performs a convolution operation on the activated data through an eighth 1×1 convolutional layer to obtain conversion data d3, and pass it to the second element-by-element addition unit;

[0143] Step C3: The second element-by-element addition unit adds the first batch of normalized data x2 and the conversion data d3 element-by-element to obtain global attention data x4.

[0144] like Figure 5 As shown: in the step 3, the feature fusion module is provided with a first feature fusion layer, a second feature fusion layer, a third feature fusion layer and a fourth feature fusion layer, wherein the first feature fusion layer is provided with a first acquisition unit and a first fusion unit, the second feature fusion layer is provided with a second acquisition unit, a second fusion unit and a third fusion unit, the third feature fusion layer is provided with a third acquisition unit, a fourth fusion unit and a fifth fusion unit, and the fourth feature fusion layer is provided with a fourth acquisition unit and a sixth fusion unit;

[0145] The feature fusion module performs feature fusion operation, and the specific steps are as follows:

[0146] Step 31: The first acquisition unit acquires the first convolution data b1 and transmits it to the first fusion unit;

[0147] The second acquisition unit acquires the second convolution data b2 and transmits it to the second fusion unit and the third fusion unit;

[0148] The third acquisition unit acquires the third convolution data b3 and passes it to the fourth fusion unit and the fifth fusion unit;

[0149] The fourth acquisition unit acquires the fourth convolution data b4 and transmits it to the fourth fusion unit and the sixth fusion unit;

[0150] Step 32: the fourth fusion unit performs feature fusion on the third convolution data b3 and the fourth convolution data b4 to obtain fourth fusion data f4, and transmits the fourth fusion data f4 to the second fusion unit and the fifth fusion unit;

[0151] The second fusion unit performs feature fusion on the second convolution data b2 and the fourth fusion data f4 to obtain second fusion data f2, and transmits the second fusion data f2 to the first fusion unit and the third fusion unit;

[0152] Step 33: the first fusion unit performs feature fusion on the first convolution data b1 and the second fusion data f2 to obtain first fusion data f1, and passes it to the third fusion unit and the third element-by-element addition unit;

[0153] The third fusion unit performs feature fusion on the second convolution data b2, the first fusion data f1 and the second fusion data f2 to obtain third fusion data f3, and passes the third fusion data f3 to the fifth fusion unit and the third element-by-element addition unit;

[0154] The fifth fusion unit performs feature fusion on the third convolution data b3, the fourth fusion data f4 and the third fusion data f3 to obtain fifth fusion data f5, and passes the fifth fusion data f5 to the sixth fusion unit and the third element-by-element addition unit;

[0155] The sixth fusion unit performs feature fusion on the fourth convolution data b4 and the fifth fusion data f5 to obtain sixth fusion data f6, and passes the sixth fusion data f6 to the third element-by-element addition unit;

[0156] Step 34: the third element-by-element adding unit adds the first fused data f1, the third fused data f3, the fifth fused data f5 and the sixth fused data f6 element-by-element to obtain fused data c.

[0157] In step 4, the classification module is provided with a first fully connected layer and a second fully connected layer;

[0158] The first fully connected layer performs a fully connected operation on the fused data c to obtain fully connected data and passes it to the second fully connected layer; the second fully connected layer performs a fully connected operation on the fully connected data to obtain a classification result.

[0159] Figure 5 In the figure, arrows ①②③ represent upsampling, arrows ④⑤⑥ represent downsampling, and arrows ⑦⑧ represent skip connections.

[0160] Taking the fusion of the intermediate feature node p3 and the output feature node p4 as an example, their output can be expressed as:

[0161]

[0162] In order to prevent numerical instability, a value e is added. O(p n )(n∈{1,2,3,4,5}) represents the output of each node. Conv(·) represents a depthwise separable convolution operation, up(·) represents an upsampling operation, and down(·) represents a downsampling operation. Feature node p3 represents an intermediate node in the top-down path, and p4 represents an output node in the bottom-up path. All other feature nodes are constructed in a similar way.

[0163] The classification result is a 1×Q vector composed of 0 and 1, and the number on each vector position in the vector is used to define an image type;

[0164] In the vector, only one vector position has a number of 1, and the image type corresponding to the vector position is the classification result.

[0165] Embodiment 2:

[0166] Example 2 is based on Example 1, and a large number of experiments are carried out on a self-built dataset to verify the performance of the adaptive attention multi-scale feature fusion network, and compare its performance with several representative CNN models.

[0167] 1. Rectal cancer T staging dataset

[0168] The dataset was obtained from the Department of Oncology, Xinqiao Hospital, Army Medical University, Chongqing, China. Patient samples and clinical data were collected after obtaining approval from the relevant ethics review committee of Xinqiao Hospital. The present invention specifically selected CT samples of patients who underwent contrast-enhanced CT scans before surgery. CT images were collected from multiple scanners. In order to ensure the consistency of the data and minimize the influence of potential confounding factors, only CT sequences with a slice interval of 1.25 mm were used for annotation. Subsequently, experienced radiologists from the Department of Radiology of Xinqiao Hospital were hired to select 5-6 slices with the most prominent rectal cancer lesions from a large number of CT images of each patient as the dataset and annotate the samples. The research focus of the present invention is the T stage of rectal cancer, which is divided into four stages: T1, T2, T3, and T4 in medical practice. The number of patients and the corresponding number of CT slices in each stage are: T1 (17 patients, 129 slices), T2 (93 patients, 888 slices), T3 (58 patients, 536 slices), and T4 (150 patients, 1537 slices). The total number of patients was 318, and the total number of CT slices was 3090.

[0169] 2. Ten-fold cross validation experimental results

[0170] The present invention is compared with more than ten mainstream convolutional neural network models, and the ten-fold cross validation results are shown in Table 1:

[0171]

[0172] Table 1

[0173] Among them, ShuffleNet stands for Reorganized Channel Network, SqueezeNet stands for Channel Squeezing Network, MobileNet stands for Mobile Network, EfficientNet stands for Efficient Network, GoogLeNet stands for Google Network, ARCNN stands for Attention Residual Convolutional Neural Network, ResNet34 stands for Deep Residual Neural Network 34, GCNet stands for Global Context Network, InceptionV3 stands for Google Network V3, ResNeXt stands for Improved Deep Residual Neural Network, PANet stands for Path Aggregation Network, Panoptic FPN stands for Panoptic Feature Pyramid Network, BDRAR stands for Bidirectional Feature Pyramid Network based on Recurrent Attention Residual Module, AlexNet stands for Alex Network, ViT stands for Visual Transformer, VGG16 stands for Visual Geometry Group 16, and AMFNet stands for Adaptive Multi-Scale Feature Fusion Network.

[0174] 3. Small sample experimental results

[0175] In medical research, the acquisition and annotation of medical images are highly complex and specialized, and often face the challenge of limited datasets. This is particularly evident in the field of cancer research, because medical image annotation is time-consuming and requires a high level of expertise. For many medical studies, obtaining datasets of sufficient size and diversity has become a daunting task, so testing with small sample data is of great significance.

[0176] The present invention designs 7 different ratios of training samples to test samples, which are 2:8, 3:7, 4:6, 5:5, 6:4, 7:3 and 8:2, thereby generating seven sets of training and test pairs with different numbers of samples. As the ratio decreases, the number of training samples gradually decreases. The experimental results are shown in Table 2:

[0177]

[0178]

[0179] Table 2

[0180] The experimental results show that the method proposed in the present invention is superior to mainstream methods in both ten-fold cross validation and small sample conditions. At the same time, the model has fewer parameters and has higher practical value than most mainstream methods. The method of the present invention combines the attention mechanism with feature fusion for the first time and uses CT images for rectal cancer T staging diagnosis. The specific advantages are as follows:

[0181] 1. The adaptive multi-branch attention module can perform effective feature extraction;

[0182] 2. Use feature fusion to perform multi-dimensional and multi-level information fusion;

[0183] 3. The number of network parameters is small, which creates conditions for deployment on hardware devices and is feasible;

[0184] 4. It can accurately predict the four T stages of rectal cancer, providing a new method for auxiliary diagnosis.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image classification and recognition method based on an adaptive multi-scale feature fusion network, characterized in that: The following steps are involved: Step 1: The image acquisition module acquires image data a and passes it to the adaptive multi-scale feature fusion network; Step 2: The feature extraction module in the adaptive multi-scale feature fusion network performs a feature extraction operation on the image data a to obtain feature data b, and passes it to the feature fusion module; Step 3: The feature fusion module performs a feature fusion operation on the feature data b to obtain fused data c, and passes it to the classification module; Step 4: The classification module classifies and identifies the fused data c to obtain a classification result.

2. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 1 is characterized in that: In step 2, the feature extraction module is provided with four adaptive multi-branch attention modules AMA, and the four adaptive multi-branch attention modules AMA have the same structure and are connected head to tail in sequence; The feature extraction module extracts features from the image data a, and the specific steps are as follows: Step 21: The first AMA1 in the feature extraction module acquires the image data a, performs feature extraction on the image data a, obtains the first feature data a1, and passes it to the second AMA2 and the first 1×1 convolutional layer; Step 22: the second AMA2 performs feature extraction on the first feature data a1 to obtain second feature data a2, and passes it to the third AMA3 and the second 1×1 convolutional layer; Step 23: the third AMA3 performs feature extraction on the second feature data a2 to obtain third feature data a3, and passes it to the fourth AMA4 and the third 1×1 convolutional layer; Step 24: the fourth AMA4 performs feature extraction on the third feature data a3 to obtain fourth feature data a4, and passes it to the fourth 1×1 convolutional layer; Step 25: The first 1×1 convolution layer performs a one-dimensional convolution operation on the first feature data a1 to obtain first convolution data b1, and passes it to the feature fusion module; The second 1×1 convolution layer performs a one-dimensional convolution operation on the second feature data a2 to obtain second convolution data b2, and passes it to the feature fusion module; The third 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain third convolution data b3, and passes it to the feature fusion module; The fourth 1×1 convolution layer performs a one-dimensional convolution operation on the third feature data a3 to obtain fourth convolution data b4, and passes it to the feature fusion module.

3. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 2 is characterized in that: In the step 21, the first AMA1 performs feature extraction on the image data a, and the specific steps are as follows: Step 211: the depth separable layer in the first AMA1 obtains the image data a, performs depth separation on the image data a, obtains depth separation data x1, and passes it to the first batch of normalization layers; Step 212: the first batch of normalization layers performs batch normalization processing on the depth separation data x1 to obtain a first batch of normalized data x2, and passes it to the convolutional attention mechanism CBAM, the global context attention mechanism GC and the fifth 1×1 convolutional layer; Step 213a: the convolutional attention mechanism CBAM performs convolutional attention adjustment on the first batch of normalized data x2 to obtain convolutional attention data x3, and passes it to the second batch of normalized layers; Step 213b: the global context attention mechanism GC performs global context attention adjustment on the first batch of normalized data x2 to obtain global attention data x4, and passes it to the third batch of normalized layers; Step 213c: the fifth 1×1 convolution layer performs a one-dimensional convolution operation on the first batch of normalized data x2 to obtain fifth convolution data x5, and passes it to the fourth batch normalization layer; Step 214a: The second batch normalization layer performs batch normalization processing on the convolution attention data x3, and then activates it through the linear rectification function ReLu to obtain the first activation data y1, and passes it to the fusion module; Step 214b: The third batch normalization layer performs batch normalization processing on the global attention data x4, and then activates it through a linear rectification function ReLu to obtain second activation data y2, and passes it to the fusion module; Step 214c: the fourth batch normalization layer performs batch normalization processing on the fifth convolution data x5, and then activates it through the layer normalization function ReLu to obtain the third activation data y3, and passes it to the fusion module; Step 215: The fusion module fuses the first activation data y1, the second activation data y2 and the third activation data y3 to obtain first feature data a1.

4. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 3 is characterized in that: In step 213a, the convolutional attention mechanism CBAM is provided with a channel attention mechanism and a spatial attention mechanism, and the convolutional attention mechanism CBAM performs a convolutional attention adjustment process as follows: The channel attention mechanism multiplies the first batch of normalized data x2 and the channel attention map Mc to obtain the channel attention data M; the spatial attention mechanism multiplies the channel attention data and the spatial attention map Ms to obtain the convolution attention data x3.

5. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 4 is characterized in that: The channel attention map Mc is obtained by the following steps: Step A1a: the first maximum pooling layer in the channel attention mechanism performs a maximum pooling operation on the first batch of normalized data x2 to obtain the first maximum pooling data c1, and passes it to the multi-layer perceptron; Step A1b: the first average pooling layer in the channel attention mechanism performs an average pooling operation on the first batch of normalized data x2 to obtain the first average pooling data c2, and passes it to the multi-layer perceptron; Step A2: the multi-layer perceptron performs multi-layer perception on the first maximum pooling data c1 and the first average pooling data c2 to obtain perception data c3, and passes it to the second maximum pooling layer and the second average pooling layer; Step A3a: the second maximum pooling layer performs a maximum pooling operation on the perception data c3 to obtain second maximum pooling data c4, and passes it to the first element-by-element addition unit; Step A3b: the second average pooling layer performs an average pooling operation on the perception data c3 to obtain second average pooling data c5, and passes it to the first element-by-element addition unit; Step A4: the first element-by-element addition unit adds the second maximum pooling data c4 and the second average pooling data c5 element-by-element, and then activates them through a sigmoid function to obtain a channel attention map Mc; The calculation equation of the channel attention map Mc is as follows: Among them, σ represents the sigmoid function, W0 and W1 represent the weights of the multi-layer perceptron, MLP represents the multi-layer perceptron, AvgPool represents average pooling, and MaxPool represents maximum pooling; F1 represents the input feature, that is, the first batch of normalized data x2; and Indicates channel information.

6. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 4 is characterized in that: The spatial attention map Ms is obtained by the following steps: Step B1: the third maximum pooling layer in the spatial attention mechanism performs a maximum pooling operation on the channel attention map Mc to obtain the third maximum pooling data s1, and passes it to the third average pooling layer; Step B2: the third average pooling layer performs an average pooling operation on the channel attention map Mc to obtain the third average pooling data s2, and passes it to the convolution layer; Step B3: the convolution layer performs a convolution operation on the third average pooling data s2 to obtain convolution data s3, and passes it to the fourth maximum pooling layer; Step B4: the fourth maximum pooling layer performs a maximum pooling operation on the convolution data s3 to obtain fourth maximum pooling data s4, and passes it to the fourth average pooling layer; Step B5: the fourth average pooling layer performs an average pooling operation on the fourth maximum pooling data s4, and then activates it through a sigmoid function to obtain a spatial attention map Ms; The calculation equation of the spatial attention map Ms is as follows: Among them, f 7×7 represents a convolution operation with a filter size of 7×7, and Represents spatial information.

7. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 3 is characterized in that: In step 213b, the global context attention mechanism GC is provided with a context modeling module and a conversion module, and the global context attention mechanism GC performs global context attention adjustment, and the specific steps are as follows: Step C1: the sixth 1×1 convolutional layer in the context modeling module performs a convolution operation on the first batch of normalized data x2, and then activates it through a sigmoid function to obtain fourth activation data d1, and then multiplies the fourth activation data d1 and the first batch of normalized data x2 element by element to obtain modeling data d2, and passes it to the conversion module; Step C2: the seventh 1×1 convolutional layer in the conversion module performs a convolution operation on the modeling data d2, and then activates the modeling data d2 through a layer normalization function ReLu, and then performs a convolution operation on the activated data through an eighth 1×1 convolutional layer to obtain conversion data d3, and pass it to the second element-by-element addition unit; Step C3: the second element-by-element adding unit adds the first batch of normalized data x2 and the conversion data d3 element-by-element to obtain global attention data x4; The specific modeling process of the global context attention mechanism GC is as follows: K=s(resize(f 1×1 (Q))) M o =Q+V Among them, Q represents the input feature map of the GC module, that is, the first batch of normalized data x2; M o represents the output feature map of the GC module, i.e., the global attention data x4; resize(·) represents the shape transformation operation, s(·) represents the softmax function, g(·) represents the layer normalization and ReLU function, represents element-by-element multiplication, f 1×1 Represents a convolution operation with a filter size of 1×1.

8. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 1 is characterized in that: In step 3, the feature fusion module is provided with a first feature fusion layer, a second feature fusion layer, a third feature fusion layer and a fourth feature fusion layer, wherein the first feature fusion layer is provided with a first acquisition unit and a first fusion unit, the second feature fusion layer is provided with a second acquisition unit, a second fusion unit and a third fusion unit, the third feature fusion layer is provided with a third acquisition unit, a fourth fusion unit and a fifth fusion unit, and the fourth feature fusion layer is provided with a fourth acquisition unit and a sixth fusion unit; The feature fusion module performs feature fusion operation, and the specific steps are as follows: Step 31: The first acquisition unit acquires the first convolution data b1 and transmits it to the first fusion unit; The second acquisition unit acquires the second convolution data b2 and transmits it to the second fusion unit and the third fusion unit; The third acquisition unit acquires the third convolution data b3 and passes it to the fourth fusion unit and the fifth fusion unit; The fourth acquisition unit acquires the fourth convolution data b4 and transmits it to the fourth fusion unit and the sixth fusion unit; Step 32: the fourth fusion unit performs feature fusion on the third convolution data b3 and the fourth convolution data b4 to obtain fourth fusion data f4, and transmits the fourth fusion data f4 to the second fusion unit and the fifth fusion unit; The second fusion unit performs feature fusion on the second convolution data b2 and the fourth fusion data f4 to obtain second fusion data f2, and transmits the second fusion data f2 to the first fusion unit and the third fusion unit; Step 33: the first fusion unit performs feature fusion on the first convolution data b1 and the second fusion data f2 to obtain first fusion data f1, and passes it to the third fusion unit and the third element-by-element addition unit; The third fusion unit performs feature fusion on the second convolution data b2, the first fusion data f1 and the second fusion data f2 to obtain third fusion data f3, and passes the third fusion data f3 to the fifth fusion unit and the third element-by-element addition unit; The fifth fusion unit performs feature fusion on the third convolution data b3, the fourth fusion data f4 and the third fusion data f3 to obtain fifth fusion data f5, and passes the fifth fusion data f5 to the sixth fusion unit and the third element-by-element addition unit; The sixth fusion unit performs feature fusion on the fourth convolution data b4 and the fifth fusion data f5 to obtain sixth fusion data f6, and passes the sixth fusion data f6 to the third element-by-element addition unit; Step 34: the third element-by-element adding unit adds the first fused data f1, the third fused data f3, the fifth fused data f5 and the sixth fused data f6 element-by-element to obtain fused data c.

9. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 1 is characterized in that: In step 4, the classification module is provided with a first fully connected layer and a second fully connected layer; The first fully connected layer performs a fully connected operation on the fused data c to obtain fully connected data and passes it to the second fully connected layer; the second fully connected layer performs a fully connected operation on the fully connected data to obtain a classification result.

10. The image classification and recognition method based on the adaptive multi-scale feature fusion network according to claim 1 or 9, characterized in that: The classification result is a 1×Q vector composed of 0 and 1, and the number on each vector position in the vector is used to define an image type; In the vector, only one vector position has a number of 1, and the image type corresponding to the vector position is the classification result.

Images