A fur image classification method, system and storage medium of multi-scale attention

Abstract

The application discloses a kind of multi-scale attention fur image classification method, system and storage medium, the application proposes multi-scale attention image classification method, accurately identifies classification animal microscopic fur image using channel and space attention, multi-scale feature extraction network, adaptive decision fusion method.First, channel attention combines the intercorrelation information between channels and feedforward convolutional neural network branch, and spatial attention combines the intercorrelation information between spatial positions and feedforward convolutional neural network branch.Then use the bottom-up transmission branch to pass the multi-channel semantic features of low layer to high layer, and then use the top-down transmission branch to fuse the high-resolution spatial features of high layer to low layer, pass the features of three different branches through feature aggregation network, and finally use the adaptive decision fusion method based on entropy to obtain the final classification result.

Description

Technical Field

[0001] This application relates to the fields of data processing and data transmission, and more specifically, to a multi-scale attention-based method, system, and storage medium for fur image classification. Background Technology

[0002] China is a major producer and consumer of animal fur, with total production and consumption increasing year by year. However, fur identification during production still relies mainly on the visual recognition of skilled workers, which cannot guarantee product stability and consistency. Furthermore, due to the fine-grained nature of animal fur microscopic images, which result in large intra-class differences and small inter-class differences in classification tasks, this classification task presents a significant challenge. Summary of the Invention

[0003] In view of the above problems, the object of the present invention is to provide a method, system and storage medium for fur image classification based on multi-scale attention.

[0004] The first aspect of this invention provides a multi-scale attention-based fur image classification method, comprising:

[0005] Get the current animal fur image data;

[0006] The current animal fur image data is sent to a preset first detection model;

[0007] Obtain the classification results information;

[0008] The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module.

[0009] In this solution, the multi-scale feature extraction module specifically comprises:

[0010] The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result.

[0011] Integrate the results from the lower level into the higher level;

[0012] Spatial attention module and channel attention module were added to the two branches.

[0013] In this solution, the spatial attention module includes:

[0014] Obtain the spatial attention module in CBAM;

[0015] The spatial attention module in the CBAM is improved according to preset rules;

[0016] The spatial attention module is obtained.

[0017] In this solution, the channel attention module includes:

[0018] Obtain the channel attention module in CBAM;

[0019] The channel attention module in the CBAM is improved according to preset rules;

[0020] The channel attention module is obtained.

[0021] In this solution, the adaptive decision fusion module includes:

[0022] The adaptive decision fusion module uses an adaptive entropy weighted decision fusion method to assign different fusion weights to the softmax outputs of different networks.

[0023] In this solution, before sending the current animal fur image data to the preset first detection model, the following steps are also included:

[0024] Obtain dataset information;

[0025] The dataset information is augmented to obtain augmented dataset information;

[0026] The augmented dataset information is used for training according to preset rules;

[0027] The preset first detection model is obtained.

[0028] A second aspect of the present invention provides a multi-scale attention fur image classification system, comprising a memory and a processor. The memory includes a multi-scale attention fur image classification method program, which, when executed by the processor, performs the following steps:

[0029] Get the current animal fur image data;

[0030] The current animal fur image data is sent to a preset first detection model;

[0031] Obtain the classification results information;

[0032] The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module.

[0033] In this solution, the multi-scale feature extraction module specifically comprises:

[0034] The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result.

[0035] Integrate the results from the lower level into the higher level;

[0036] Spatial attention module and channel attention module were added to the two branches.

[0037] In this solution, the spatial attention module includes:

[0038] Obtain the spatial attention module in CBAM;

[0039] The spatial attention module in the CBAM is improved according to preset rules;

[0040] The spatial attention module is obtained.

[0041] In this solution, the channel attention module includes:

[0042] Obtain the channel attention module in CBAM;

[0043] The channel attention module in the CBAM is improved according to preset rules;

[0044] The channel attention module is obtained.

[0045] In this solution, the adaptive decision fusion module includes:

[0046] The adaptive decision fusion module uses an adaptive entropy weighted decision fusion method to assign different fusion weights to the softmax outputs of different networks.

[0047] In this solution, before sending the current animal fur image data to the preset first detection model, the following steps are also included:

[0048] Obtain dataset information;

[0049] The dataset information is augmented to obtain augmented dataset information;

[0050] The augmented dataset information is used for training according to preset rules;

[0051] The preset first detection model is obtained.

[0052] A third aspect of the present invention provides a computer-readable storage medium comprising a multi-scale attention fur image classification method program, wherein when the multi-scale attention fur image classification method program is executed by a processor, it implements the steps of the multi-scale attention fur image classification method as described in any of the preceding claims.

[0053] This invention discloses a multi-scale attention-based method, system, and storage medium for fur image classification. The proposed method utilizes channel and spatial attention, a multi-scale feature extraction network, and an adaptive decision fusion method to accurately identify and classify animal microscopic fur images. First, channel attention combines cross-correlation between channels with a feedforward convolutional neural network branch, while spatial attention combines cross-correlation between spatial locations with a feedforward convolutional neural network branch. Then, a bottom-up propagation branch is used to pass low-level multi-channel semantic features to higher levels. Next, a top-down propagation branch is used to fuse high-resolution spatial features from higher levels back to lower levels. Features from the three different branches are then fused through a feature aggregation network. Finally, an entropy-based adaptive decision fusion method is used to obtain the final classification result. Attached Figure Description

[0054] Figure 1 A flowchart of a multi-scale attention-based fur image classification method according to the present invention is shown;

[0055] Figure 2 A flowchart of an improved spatial attention module method according to the present invention is shown;

[0056] Figure 3 A flowchart of an improved method for a channel attention module according to the present invention is shown;

[0057] Figure 4 A block diagram of a multi-scale attention fur image classification system according to the present invention is shown;

[0058] Figure 5 A schematic diagram of the overall architecture of a network according to the present invention is shown;

[0059] Figure 6 A schematic diagram of a multi-scale attention module according to the present invention is shown;

[0060] Figure 7 A schematic diagram of the structure of a spatial attention module according to the present invention is shown;

[0061] Figure 8 A schematic diagram of the structure of a channel attention module according to the present invention is shown;

[0062] Figure 9 A schematic diagram of the structure of a local feature enhancement module according to the present invention is shown;

[0063] Figure 10 A schematic diagram of the structure of an adaptive weight fusion module according to the present invention is shown;

[0064] Figure 11 A schematic diagram illustrating a dataset data source according to the present invention is shown;

[0065] Figure 12 This diagram illustrates a dataset after processing according to the present invention.

[0066] Figure 13 The diagram shows the effect of a category classification method according to the present invention. Detailed Implementation

[0067] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0068] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0069] Figure 1 A flowchart of a multi-scale attention-based fur image classification method according to the present invention is shown.

[0070] like Figure 1 As shown, this invention discloses a multi-scale attention-based fur image classification method, comprising:

[0071] S102, Obtain the current animal fur image data;

[0072] S104, The current animal fur image data is sent to the preset first detection model;

[0073] S106, obtain the classification result information;

[0074] The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module.

[0075] According to embodiments of the present invention, common CNN networks such as VGG

[23] , ResNet

[24] , and DenseNet

[25] perform well in ordinary image classification tasks, but there is still a lot of room for optimization and improvement for fine-grained image classification tasks. The present invention proposes a weakly supervised network. The first detection model is an image classification method based on ResNet50, which does not require manual annotation. While taking into account classification efficiency, it has excellent classification accuracy. In addition, since the characteristics of microscopic fur images are that the intra-class differences are large and the inter-class differences are small, the distinguishable features are less fine-grained than those of traditional images. Therefore, segmenting the image before recognition is equivalent to enhancing the local fine-grained features. Slicing the large image into blocks for recognition can also reduce the classification error rate. After image is sliced, the results of the image patches need to be fused. The traditional fusion method is direct voting, which means that the confidence of all image patches is regarded as 1. Different image patches have different backgrounds, interference, and quality, which can easily lead to low accuracy of the final classification result. Therefore, this invention proposes an adaptive fusion decision method based on information entropy. This method adaptively adjusts the weights of the classification results of each category based on the network. When the model cannot distinguish the category of an image patch well, it will reduce the confidence of the result of that image patch to avoid affecting the result.

[0076] The overall architecture of the network is as follows Figure 5 As shown, it mainly consists of two phases: the training phase and the testing phase.

[0077] (1) The training phase mainly involves training the network model, which consists of a feature extraction base network, a multi-scale feature extraction network, a feature aggregation network, and an adaptive decision fusion module.

[0078] (1.1) First, in the feature extraction base network stage, ResNet50 is used as the backbone network for feature extraction, and a pre-trained model on ImageNet is used for network parameter initialization. In the network, the last two layers of the original ResNet50, namely the average pooling layer and the fully connected layer, are removed to output the feature flow. We extract multi-scale features from layers conv2_10, conv3_12, and conv4_18, with feature dimensions of 28x28x512, 14x14x1024, and 7x7x2048, respectively, as the inputs to the three branches of the multi-scale attention module.

[0079] (1.2) In the multi-scale attention network stage, three feature branches that integrate texture and semantics from different layers are generated by channel attention module and bottom-up branch, and spatial attention module and top-down branch, which enhances feature expression ability and improves classification accuracy.

[0080] (1.3) In the feature aggregation stage, since the three branches will have some redundant features as they are continuously fused, the three branches generated in the previous stage are aggregated by the feature aggregation module, the features are then concatenated, and finally the result label is output through the fully connected layer.

[0081] (2) The testing phase mainly involves first cutting the image into blocks, passing each block through the trained CNN network, which will output multiple softmax probability matrices, and then outputting the final label results through the entropy-based adaptive decision fusion module.

[0082] According to an embodiment of the present invention, the multi-scale feature extraction module specifically comprises:

[0083] The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result.

[0084] Integrate the results from the lower level into the higher level;

[0085] Spatial attention module and channel attention module were added to the two branches.

[0086] It should be noted that traditional CNN networks do not utilize the multi-scale feature information extracted at different stages, but only use the feature map of the last convolutional layer for classification. This method is only suitable for images with a single target. However, the microscopic fur images studied in this invention are characterized by the locality of features, which differs from the targets on traditional ImageNet. A combination of local texture and global features is needed for better classification; therefore, the texture details of the image are also very important. Traditional FPN networks [reference] simply sum the feature maps of different convolutional layers element-wise, merely fusing high-level semantic features into the lower layers without forming multi-scale, multi-level features. Traditional Unet networks [reference] simply concatenate the feature maps of different convolutional layers through channels to form multi-scale features without fully utilizing high-level semantic information. The method of this invention combines the two approaches: first, element-wise summation is used to fuse high-level semantic features into the lower layers, and then channel concatenation is used to fuse the results from the lower layers into the higher layers. Spatial and channel attention modules are also added to the two branches to further enhance important features. Therefore, the output is a multi-scale, multi-level feature map that fully integrates spatial and semantic features.

[0087] like Figure 6As shown, the multi-scale attention module contains two propagation branches: bottom-up and top-down feature propagation channels. The bottom-up propagation consists of the CA module, the CBR module, addition, and upsampling operations. The CA module extracts channel attention from the feature map, strengthening the semantic features of more important channels. Upsampling interpolates the image to increase resolution, allowing for fusion with features from other channels of higher resolution. The CBR module combines Conv1x1, BatchNorm, and ReLU. Conv1x1 reduces channel dimensionality, BatchNorm accelerates model convergence, effectively avoids gradient vanishing, and improves generalization ability. ReLU introduces non-linearity to enhance model expressiveness.

[0088] For ease of calculation, the number of feature channels in each branch is first reduced to 256 before this module to facilitate computation. The entire branch process from bottom to top is as follows: First, feature map F3∈R C×H×W After CA channel attention, important channel features in the feature map are enhanced. Upsampling is then performed to adjust the size to C2. This upsampling uses interpolation or learning-based methods. The feature map is then element-wise added to C2 and fused, before being fed into the CBR module for feature smoothing. This incorporates the features of C3 into C2. The fused features of C3 and C2 are then further upsampled to the feature map size of C1 and added to C1 for fusion. The formula is as follows:

[0089] P3 = CA(F3)

[0090] P2 = CBR(UP(P2) + F2)

[0091] P1 = CBR(UP(P2) + SA(F1))

[0092] P3 = CA(F3)

[0093] P2 = CBR(UP(P2) + F2)

[0094] P1 = CBR(UP(P2) + SA(F1))

[0095] CA stands for Channel Attention Module, CBR is a combination of Conv1x1, BatchNorm and ReLU, UP is the upsampling operation, and SA is the Spatial Attention Module.

[0096] The top-down propagation consists of the SA module, the CBR module, downsampling, and concatenation operations. First, C1 is sent to the SA module to extract spatial attention, which enhances spatial features. Then, it is further concatenated and fused with the upsampled features. Concatenation fusion differs from additive fusion; the former combines features to create diversity, while the latter simply enhances features. Finally, the fused features are passed through the CBR module, generating the C1' feature. This allows the lower-level channel attention features to be fused into the higher-level features. Ultimately, three feature maps of different scales—C1', C2', and C3'—are generated, representing multi-level, multi-scale features.

[0097] N3 = CBR(Concat(DO(N2)+P3))

[0098] N2 = CBR(Concat(DO(N1) + P2))

[0099] N1 = P1, where DO is the downsampling operation and Concat is the concatenation operation.

[0100] According to an embodiment of the present invention, the spatial attention module includes:

[0101] S202, Obtain the spatial attention module in CBAM;

[0102] S204, Improve the spatial attention module in the CBAM according to preset rules;

[0103] S206, the spatial attention module is obtained.

[0104] It should be noted that, as Figure 7 As shown, the spatial attention module of this invention is an improvement upon the spatial attention module in CBAM. The original module first concatenates the results of MaxPool and AvgPool, then performs a convolution, and finally generates spatial attention weights through an activation function. The problem with the spatial attention module of CBAM is that it does not use an MLP to increase the non-linear mapping relationship after convolution, resulting in a loss of feature expressiveness. The module of this invention adds an MLP network at the end, which consists of two fully connected (FC) layers.

[0105] The specific process is as follows: First, given the feature map F∈R C×H×W Using global average pooling and global max pooling, the spatial information is compressed and the dimension is eliminated to obtain: F c =F·F r (F)+F and The formula is as follows

[0106]

[0107]

[0108] Where MaxPool represents global average pooling and AvgPool represents global max pooling.

[0109] Then concatenate the two results to get F. concat ∈R 2C×1×1 As shown in the following formula

[0110]

[0111] Concat represents the concatenation operation based on channels.

[0112] Since the concatenated feature map has 2C channels, a convolutional layer is needed to change the channel dimension to C, as shown in the following formula.

[0113] F'=Conv(F concat )

[0114] Where Conv represents a 1x1 channel convolution operation.

[0115] Finally, F' is forwarded to an MLP (Multilayer Perceptron) network to compress the dimensionality, which is equivalent to using two fully connected (FC) layers to generate the channel attention feature map F. r ∈R C×1×1 Finally, the feature map is activated by the Sigmoid function. After the activation function, each weight value of the feature map will be in the range of (0,1).

[0116] F r =σ(MLP(F'))

[0117] Where σ represents the Sigmoid activation function.

[0118] The final formula for calculating the channel attention weights is as follows:

[0119] F r (F)=σ(MLP(Conv(Concat(Avg(F),Max(F)))))

[0120] Where F r (F)∈R C×1×1 Represents the channel weight matrix

[0121] After obtaining the attention weights for the channel features, the weights are multiplied by the original feature map to obtain the attention map. Finally, a residual connection is used to superimpose the attention map onto the original feature map, generating the final feature map. The formula is as follows:

[0122] F c =F·F r(F)+F

[0123] Where F c F is the resulting feature map, and F is the original feature map. r (F) represents the channel attention weight.

[0124] According to an embodiment of the present invention, the channel attention module includes:

[0125] S302, Obtain the channel attention module in CBAM;

[0126] S304, Improve the channel attention module in the CBAM according to preset rules;

[0127] S306, the channel attention module is obtained.

[0128] It should be noted that, as Figure 8 The channel attention module of this invention is an improvement upon the channel attention module in CBAM. The original module first passes the results of MaxPool and AvgPool through a shared MLP network, then sums the results and generates channel attention weights through an activation function. The problem with channel attention in CBAM is that summing the results of two pooling operations may result in the loss of spatial information. This invention addresses this by concatenating the results before passing them through the MLP, then passing them through a channel convolutional layer, essentially performing attention-based feature selection. This preserves useful information and reduces information loss. Finally, a fully connected (FC) layer maps the dimension to the channel dimension of the feature map.

[0129] The specific process is as follows: First, given the feature map F∈R C×H×W Using global average pooling and global max pooling, the spatial information is compressed and the dimensionality is eliminated to obtain: and The formula is as follows

[0130]

[0131]

[0132] Where MaxPool represents global average pooling and AvgPool represents global max pooling.

[0133] Then concatenate the two results to get F. concat =R 2C×1×1 As shown in the following formula

[0134]

[0135] Where concat represents the concatenation operation by channel.

[0136] Since the concatenated feature map has 2C channels, a single channel convolutional layer Con1x1 is needed to change the channel dimension to C, as shown in the following formula.

[0137] F′=Conv(F concat )

[0138] Where Conv represents a 1x1 channel convolution operation.

[0139] Finally, F′ is forwarded to an MLP (Multilayer Perceptron) network to compress the dimensionality, which is equivalent to using two fully connected (FC) layers to generate the channel attention feature map F. r ∈R C×1×1 Finally, the feature map is activated by the Sigmoid function. After the activation function, each weight value of the feature map will be in the range of (0,1).

[0140] F r =σ(MLP(F′))

[0141] Where σ represents the Sigmoid activation function.

[0142] The final formula for calculating the channel attention weights is as follows:

[0143] F r (F)=σ(MLP(Conv(Concat(Avg(F),Max(F))))

[0144] Where F r (F)∈R C×1×1 Represents the channel weight matrix

[0145] After obtaining the attention weights for the channel features, the weights are multiplied by the original feature map to obtain the attention map. Finally, a residual connection is used to superimpose the attention map onto the original feature map, generating the final feature map. The formula is as follows:

[0146] F c =F·F r (F)+F

[0147] Where F c F is the resulting feature map, and F is the original feature map. r (F) represents the channel attention weight.

[0148] Local feature enhancement module, such as Figure 9 As shown, after the multi-scale feature extraction module, the feature dimensions of the three branches are high and the features are redundant. Therefore, the feature aggregation module is needed to further filter and enhance the important features to reduce feature redundancy and computation.

[0149] This module consists of channel convolution, pooling, unrolling, and fully connected (FC) layers. First, pooling reduces spatial resolution to increase the receptive field of features. Then, all channel features of each pixel are directly unrolled, followed by channel convolution and FC layers. This is equivalent to image feature-based attention, preserving important feature information from the feature map and removing redundant features to improve classification accuracy. The entire process can be represented by the following formula.

[0150] F=FC(Conv(Flattern(Pooling(x))))

[0151] Where x represents the input feature map, Pooling represents average pooling, Conv represents a channel convolution with a kernel size of 1, FC represents a fully connected layer, and Flattern represents expanding the features to eliminate spatial dimensions.

[0152] According to an embodiment of the present invention, the adaptive decision fusion module includes:

[0153] The adaptive decision fusion module uses an adaptive entropy weighted decision fusion method to assign different fusion weights to the softmax outputs of different networks.

[0154] It should be noted that during the testing phase, since there are multiple image patches for the final decision fusion, a feature fusion method is needed to fuse the results, such as... Figure 10 As shown, each image patch has its own softmax probability matrix, which describes the probability of each class as perceived by the model. The higher the probability of a certain class, the more likely the model thinks the image belongs to that class. However, it is also possible that two classes are very similar, in which case the model may consider the probability of both classes to be high. Therefore, the decision result is not very reliable in this case, so it should be given a lower weight to make its confidence level lower.

[0155] The maximum output probability decision fusion method is the simplest decision fusion method. It determines the final class of the input image by calculating the maximum probability of the outputs of different sub-networks. If the maximum output probability of each classifier falls within the same class, the decision result remains unchanged. However, as the probabilities change, the likelihood of the decision varies significantly. This method does not consider the weights assigned to each classifier during decision fusion; however, the fusion weight of each classifier determines the classification performance. Different sub-networks extract different features, and the samples have a certain degree of randomness, so a suitable decision method needs to be developed to obtain optimal classification performance.

[0156] An adaptive entropy-weighted decision fusion method is used to assign different fusion weights to the softmax outputs of different networks, further improving the classification accuracy of the algorithm while ensuring better model stability and robustness. This invention involves multiple image patches participating in the decision-making process; therefore, the fusion decision is obtained by adaptively entropy-weighted decision fusion of the probability values ​​from multiple image outputs to arrive at the final classification result. The specific adaptive entropy-weighted decision fusion algorithm is as follows.

[0157] When multiple softmax functions are input in parallel, the probability output matrix can be obtained as follows:

[0158]

[0159] Where m is the number of image patches, n is the number of categories, and p ij (x) represents the probability of the i-th image patch for class j. The column index of the highest probability in each row is the predicted class of the sample by the Softmax function for each image. The characteristics of the probability values ​​of the input sample belonging to each class will affect the classification accuracy. Specifically, the smaller the difference between the probability values, the greater the uncertainty of the classification; if the largest probability value is more different from the other probability values, the classification result is more reliable. Therefore, this invention introduces the information entropy H as shown in equation (2). i (x) is used to characterize the uncertainty of the input sample being classified by the Softmax function of the i-th sub-network.

[0160]

[0161] Where, p ij (x) represents the probability that the th subnetwork's Softmax function classifies the input sample as belonging to class . If the information entropy of a subnetwork's Softmax function is higher, the classification uncertainty is higher, and the Softmax function's ability to classify the input sample is worse. Therefore, the fusion weights of the Softmax function for the input sample are smaller, and vice versa. Thus, the adaptive fusion weights calculation formula for the Softmax function is:

[0162]

[0163] Therefore, the matrix is

[0164]

[0165] The label of the maximum value of the weighted sum of P(x) is the result of decision fusion, as shown in the formula.

[0166]

[0167] The adaptive entropy-weighted decision fusion algorithm fully considers the softmax function of different image patches and adaptively assigns more reasonable fusion weights to different input images, reducing the weight of image patches with high softmax matrix entropy.

[0168] Increase the weights of image patches with low softmax matrix entropy to improve classification accuracy.

[0169] According to an embodiment of the present invention, before sending the current animal fur image data to a preset first detection model, the method further includes:

[0170] Obtain dataset information;

[0171] The dataset information is augmented to obtain augmented dataset information;

[0172] The augmented dataset information is used for training according to preset rules;

[0173] The preset first detection model is obtained.

[0174] It should be noted that the loss function of this model uses the multi-class cross-entropy loss from the training phase to optimize the network.

[0175]

[0176] Where N represents the number of samples, i represents the sample index, c represents the class, M represents the number of classes, and p ic y represents the predicted probability that the observed sample belongs to class c. ic It is a sign function; it takes the value 1 if the true class of the sample is equal to c, and 0 otherwise.

[0177] like Figure 11 The dataset consists primarily of 24 categories of animal fur: mink, kangaroo, sheep, fox, weasel, sable, cow, American sable, sable, beaver, marmot, horse, chinchilla, stone marten, raccoon, fishing mink, raccoon, otter, coypu, gray squirrel, muskrat, raccoon dog, squirrel, and rabbit. The unsegmented dataset of 24 animal fur categories shows strong similarity in the fur of otters, raccoons, squirrels, and rabbits, while raccoon dog and horsehair also exhibit high similarity. Therefore, this dataset meets the characteristics of fine-grained image datasets, with large intra-class differences and small inter-class differences. Furthermore, key features are hidden in local image features (such as whether the fur edges have barbs, or whether the central part of the fur is distinctive). This dataset can be used for subsequent experiments. However, because the dataset was captured using a microscope, some parts contain significant noise, resulting in a low signal-to-noise ratio. This portion of the data may be discarded in subsequent processing.

[0178] Since the initial dataset contained only about 100 images per class with a resolution of 768*576, image segmentation was used to further expand the dataset and improve recognition performance. This involved dividing each image into smaller segments and removing images containing only the background. The final dataset consisted of 1500 images per class, for a total of 24 classes.

[0179] Because the background of images under a microscope is related to lighting conditions, it is difficult to guarantee that all images have the same background. Since images are segmented, the background in some images may occupy the majority of the image information. Therefore, the intensity of the background color also becomes a factor that interferes with image classification accuracy. Because the original microscopic images of animal fur contain multiple hairs, and the images of hair in different parts of the body vary greatly, with some images having many blank areas, this invention slices the dataset, dividing the animal fur images into 3×3 sub-images. This not only enhances the features in the images and improves image quality, but also expands the dataset, which is beneficial for improving classification accuracy.

[0180] Edge pixel detection is used to examine pixels at the edges of an image. Since fur, which exists only in a small area, is often excessively noisy and can interfere with the classification of other image categories, we remove it. We detect pixel values ​​in the outer 20% region of the image starting from the four borders, and each new pixel value is denoted as K. i Record K i The number, if K i If the number of occurrences is less than the threshold, the image is removed. The specific expression is as follows:

[0181]

[0182] After sub-image segmentation, the detailed features of the image are more clearly displayed. At the same time, the differences between different fur types are also clearer, and the fur features of different categories can be more easily distinguished.

[0183] Processed image as follows Figure 12 As shown, the advantages of this processing are that it expands the dataset and enhances the class features of the dataset. Using the processed image dataset can amplify image detail information in a single image, which is beneficial for subsequent network feature extraction.

[0184] According to an embodiment of the present invention, data augmentation processing is performed on the dataset information to obtain augmented dataset information, including:

[0185] Data augmentation processing of the dataset information includes rotation, flipping, scaling, translation, random cropping, changing brightness, changing saturation, and changing hue.

[0186] It should be noted that, to avoid underfitting or overfitting due to image sample quality and dataset diversity, preprocessing methods such as rotation, flipping, scaling, translation, random cropping, brightness adjustment, saturation adjustment, and hue adjustment are employed. For flipping, each training image is flipped horizontally or vertically with a 50% probability, allowing the network to fully observe image changes during training and enhancing its robustness. For rotation, a small 10-degree rotation is performed around the image origin. In the translation method, the image is slightly shifted vertically and horizontally to enhance the network's robustness. In random cropping, a portion of the image is cropped and enlarged to a size of 224*224, allowing for the extraction of finer-grained features, which is more suitable for the microscopic images studied in this invention. In the brightness, saturation, and hue adjustment methods, these three attributes are randomly changed within a range of 1 to 1.35. Within this range, the image's visibility remains unchanged, further enhancing the model's robustness.

[0187] According to an embodiment of the present invention, it further includes:

[0188] The dataset includes a training set, a validation set, and a test set.

[0189] It should be noted that the experiment used a self-made dataset and a backbone network pre-trained on ImageNet. The training and validation sets were used for network training, and the test set was used for network testing, with a ratio of 8:1:1. The operating environment was Python 3.8 and PyTorch 1.7. The image enhancement methods used included random cropping, random changes to image brightness, saturation, and hue (ranging from 1 to 1.35 of the original image), horizontal flipping, and vertical flipping. Stochastic gradient descent (SGD) with momentum was chosen as the optimizer, employing a cosine roll-off learning rate decay strategy with an initial learning rate of 0.01. The training consisted of 50 epochs with a batch size of 32 and input images of 224*224 pixels. The server was running Ubuntu 18.04, with an i9 10900K CPU, a 3090 graphics card with 24GB of VRAM, and 128GB of RAM.

[0190] According to an embodiment of the present invention, it further includes:

[0191] Ablation experiment.

[0192] It should be noted that, firstly, this invention uses image segmentation to expand the dataset, increasing the original dataset of 100 images per class to 1500 images per class. The effect before and after using this method is shown in the figure.

[0193] Method Acc ResNet50 0.747 ResNet50+Pieces 0.802

[0194] As can be seen, manipulating the dataset can improve the recognition results to some extent, which can be considered a form of image enhancement. All subsequent methods will use this method and will not be elaborated further.

[0195] Method Acc ResNet50 0.802 ResNet50+SENet 0.832 ResNet50+CBAM 0.835 ResNet50+ Attention Module 0.844

[0196] The table compares existing attention modules and shows that the attention module of this invention performs better than both SENet and CBAM.

[0197] Method Acc ResNet50 0.802 +Multi-scale spatial attention module 0.837 +Multi-scale attention module 0.844 +Adaptive Fusion Decision Module 0.863

[0198] As shown in the table, the accuracy of the original ResNet50 network is only 0.802. After adding the multi-scale attention module, the model can extract multi-scale features including texture and semantic features with an accuracy of 0.855. After adding the feature aggregation module, the accuracy is 0.863. After adding the adaptive fusion decision module, the results of multiple image patches are added with adaptive weights and make joint decisions, and the accuracy can reach 0.889. It can be seen that the improved method has good reliability in recognition performance.

[0199] According to an embodiment of the present invention, it further includes:

[0200] Comparative experiment.

[0201] It should be noted that the parameter comparison involves comparing the multi-scale attention module with FPN and Unet structures, and also comparing the results of different parameters in classic network models such as VGG (e.g., hyperparameter sizes like patch size in VIT), loss curves, accuracy curves, F1 score curves, confusion matrices, Gradcam plots (visualization results), including those after adding various modules, as well as comparisons of class classification performance with other algorithms, such as... Figure 13 As shown.

[0202] Algorithm Acc ResNet50 0.802 ResNet50+CBAM 0.835 MobileNetV3-Small 0.738 EfficientNetV2-S 0.863 MobileViT-X 0.773 Vision Transformer 0.852 ResNet101 0.854 Ours 0.863

[0203] As can be seen, the original ResNet50 network has an accuracy of only 0.802. Adding the CBAM attention module to the original ResNet50 network improves the accuracy to 0.835. The EfficientNetV2-S model, proposed in 2021, is a network with faster training speed, fewer parameters, and a progressive learning strategy [citation needed], achieving an accuracy of 0.837. MobileViT-X, also proposed in 2021, uses MobileNetV3 as its backbone network and a hybrid architecture of CNN and Transformer, achieving an accuracy of 0.773. However, it has a very small number of parameters, resulting in fast training and inference speeds. ResNet101 is an upgraded version of ResNet50, increasing network depth and parameters, achieving an accuracy of 0.854. The algorithm of this invention achieves an accuracy of 0.863, outperforming other methods and demonstrating its effectiveness.

[0204] Figure 4 A block diagram of a multi-scale attention fur image classification system according to the present invention is shown.

[0205] like Figure 4 As shown, a second aspect of the present invention provides a multi-scale attention fur image classification system 4, including a memory 41 and a processor 42. The memory includes a multi-scale attention fur image classification method program, which, when executed by the processor, performs the following steps:

[0206] Get the current animal fur image data;

[0207] The current animal fur image data is sent to a preset first detection model;

[0208] Obtain the classification results information;

[0209] The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module.

[0210] According to embodiments of the present invention, common CNN networks such as VGG

[23] , ResNet

[24] , and DenseNet

[25] perform well in ordinary image classification tasks, but there is still a lot of room for optimization and improvement for fine-grained image classification tasks. The present invention proposes a weakly supervised network. The first detection model is an image classification method based on ResNet50, which does not require manual annotation. While taking into account classification efficiency, it has excellent classification accuracy. In addition, since the characteristics of microscopic fur images are that the intra-class differences are large and the inter-class differences are small, the distinguishable features are less fine-grained than those of traditional images. Therefore, segmenting the image before recognition is equivalent to enhancing the local fine-grained features. Slicing the large image into blocks for recognition can also reduce the classification error rate. After image is sliced, the results of the image patches need to be fused. The traditional fusion method is direct voting, which means that the confidence of all image patches is regarded as 1. Different image patches have different backgrounds, interference, and quality, which can easily lead to low accuracy of the final classification result. Therefore, this invention proposes an adaptive fusion decision method based on information entropy. This method adaptively adjusts the weights of the classification results of each category based on the network. When the model cannot distinguish the category of an image patch well, it will reduce the confidence of the result of that image patch to avoid affecting the result.

[0211] The overall architecture of the network is as follows Figure 5 As shown, it mainly consists of two phases: the training phase and the testing phase.

[0212] (1) The training phase mainly involves training the network model, which consists of a feature extraction base network, a multi-scale feature extraction network, a feature aggregation network, and an adaptive decision fusion module.

[0213] (1.1) First, in the feature extraction base network stage, ResNet50 is used as the backbone network for feature extraction, and a pre-trained model on ImageNet is used for network parameter initialization. In the network, the last two layers of the original ResNet50, namely the average pooling layer and the fully connected layer, are removed to output the feature flow. We extract multi-scale features from layers conv2_10, conv3_12, and conv4_18, with feature dimensions of 28x28x512, 14x14x1024, and 7x7x2048, respectively, as the inputs to the three branches of the multi-scale attention module.

[0214] (1.2) In the multi-scale attention network stage, three feature branches that integrate texture and semantics from different layers are generated by channel attention module and bottom-up branch, and spatial attention module and top-down branch, which enhances feature expression ability and improves classification accuracy.

[0215] (1.3) In the feature aggregation stage, since the three branches will have some redundant features as they are continuously fused, the three branches generated in the previous stage are aggregated by the feature aggregation module, the features are then concatenated, and finally the result label is output through the fully connected layer.

[0216] (2) The testing phase mainly involves first cutting the image into blocks, passing each block through the trained CNN network, which will output multiple softmax probability matrices, and then outputting the final label results through the entropy-based adaptive decision fusion module.

[0217] According to an embodiment of the present invention, the multi-scale feature extraction module specifically comprises:

[0218] The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result.

[0219] Integrate the results from the lower level into the higher level;

[0220] Spatial attention module and channel attention module were added to the two branches.

[0221] It should be noted that traditional CNN networks do not utilize the multi-scale feature information extracted at different stages, but only use the feature map of the last convolutional layer for classification. This method is only suitable for images with a single target. However, the microscopic fur images studied in this invention are characterized by the locality of features, which differs from the targets on traditional ImageNet. A combination of local texture and global features is needed for better classification; therefore, the texture details of the image are also very important. Traditional FPN networks [reference] simply sum the feature maps of different convolutional layers element-wise, merely fusing high-level semantic features into the lower layers without forming multi-scale, multi-level features. Traditional Unet networks [reference] simply concatenate the feature maps of different convolutional layers through channels to form multi-scale features without fully utilizing high-level semantic information. The method of this invention combines the two approaches: first, element-wise summation is used to fuse high-level semantic features into the lower layers, and then channel concatenation is used to fuse the results from the lower layers into the higher layers. Spatial and channel attention modules are also added to the two branches to further enhance important features. Therefore, the output is a multi-scale, multi-level feature map that fully integrates spatial and semantic features.

[0222] like Figure 6As shown, the multi-scale attention module contains two propagation branches: bottom-up and top-down feature propagation channels. The bottom-up propagation consists of the CA module, the CBR module, addition, and upsampling operations. The CA module extracts channel attention from the feature map, strengthening the semantic features of more important channels. Upsampling interpolates the image to increase resolution, allowing for fusion with features from other channels of higher resolution. The CBR module combines Conv1x1, BatchNorm, and ReLU. Conv1x1 reduces channel dimensionality, BatchNorm accelerates model convergence, effectively avoids gradient vanishing, and improves generalization ability. ReLU introduces non-linearity to enhance model expressiveness.

[0223] For ease of calculation, the number of feature channels in each branch is first reduced to 256 before this module to facilitate computation. The entire branch process from bottom to top is as follows: First, feature map F3∈R C×H×W After CA channel attention, important channel features in the feature map are enhanced. Upsampling is then performed to adjust the size to C2. This upsampling uses interpolation or learning-based methods. The feature map is then element-wise added to C2 and fused, before being fed into the CBR module for feature smoothing. This incorporates the features of C3 into C2. The fused features of C3 and C2 are then further upsampled to the feature map size of C1 and added to C1 for fusion. The formula is as follows:

[0224] P3 = CA(F3)

[0225] P2 = CBR(UP(P2) + F2)

[0226] P1 = CBR(UP(P2) + SA(F1))

[0227] P3 = CA(F3)

[0228] P2 = CBR(UP(P2) + F2)

[0229] P1 = CBR(UP(P2) + SA(F1))

[0230] CA stands for Channel Attention Module, CBR is a combination of Conv1x1, BatchNorm and ReLU, UP is the upsampling operation, and SA is the Spatial Attention Module.

[0231] The top-down propagation consists of the SA module, the CBR module, downsampling, and concatenation operations. First, C1 is sent to the SA module to extract spatial attention, which enhances spatial features. Then, it is further concatenated and fused with the upsampled features. Concatenation fusion differs from additive fusion; the former combines features to create diversity, while the latter simply enhances features. Finally, the fused features are passed through the CBR module, generating the C1' feature. This allows the lower-level channel attention features to be fused into the higher-level features. Ultimately, three feature maps of different scales—C1', C2', and C3'—are generated, representing multi-level, multi-scale features.

[0232] N3 = CBR(Concat(DO(N2)+P3))

[0233] N2 = CBR(Concat(DO(N1) + P2))

[0234] N1 = P1

[0235] Where DO is the downsampling operation and Concat is the concatenation operation.

[0236] According to an embodiment of the present invention, the spatial attention module includes:

[0237] Obtain the spatial attention module in CBAM;

[0238] The spatial attention module in the CBAM is improved according to preset rules;

[0239] The spatial attention module is obtained.

[0240] It should be noted that, as Figure 7 As shown, the spatial attention module of this invention is an improvement upon the spatial attention module in CBAM. The original module first concatenates the results of MaxPool and AvgPool, then performs a convolution, and finally generates spatial attention weights through an activation function. The problem with the spatial attention module of CBAM is that it does not use an MLP to increase the non-linear mapping relationship after convolution, resulting in a loss of feature expressiveness. The module of this invention adds an MLP network at the end, which consists of two fully connected (FC) layers.

[0241] The specific process is as follows: First, given the feature map F∈R C×H×W Using global average pooling and global max pooling, the spatial information is compressed and the dimension is eliminated to obtain: F c =F·F r (F)+F and The formula is as follows

[0242]

[0243]

[0244] Where MaxPool represents global average pooling and AvgPool represents global max pooling.

[0245] Then concatenate the two results to get F. concat ∈R 2C×1×1 As shown in the following formula

[0246]

[0247] Concat represents the concatenation operation based on channels.

[0248] Since the concatenated feature map has 2C channels, a convolutional layer is needed to change the channel dimension to C, as shown in the following formula.

[0249] F'=Conv(F concat )

[0250] Where Conv represents a 1x1 channel convolution operation.

[0251] Finally, F' is forwarded to an MLP (Multilayer Perceptron) network to compress the dimensionality, which is equivalent to using two fully connected (FC) layers to generate the channel attention feature map F. r ∈R C×1×1 Finally, the feature map is activated by the Sigmoid function. After the activation function, each weight value of the feature map will be in the range of (0,1).

[0252] F r =σ(MLP(F'))

[0253] Where σ represents the Sigmoid activation function.

[0254] The final formula for calculating the channel attention weights is as follows:

[0255] F r (F)=σ(MLP(Conv(Concat(Avg(F),Max(F)))))

[0256] Where F r (F)∈R C×1×1 Represents the channel weight matrix

[0257] After obtaining the attention weights for the channel features, the weights are multiplied by the original feature map to obtain the attention map. Finally, a residual connection is used to superimpose the attention map onto the original feature map, generating the final feature map. The formula is as follows:

[0258] F c =F·Fr (F)+F

[0259] Where F c F is the resulting feature map, and F is the original feature map. r (F) represents the channel attention weight.

[0260] According to an embodiment of the present invention, the channel attention module includes:

[0261] Obtain the channel attention module in CBAM;

[0262] The channel attention module in the CBAM is improved according to preset rules;

[0263] The channel attention module is obtained.

[0264] It should be noted that, as Figure 8 The channel attention module of this invention is an improvement upon the channel attention module in CBAM. The original module first passes the results of MaxPool and AvgPool through a shared MLP network, then sums the results and generates channel attention weights through an activation function. The problem with channel attention in CBAM is that summing the results of two pooling operations may result in the loss of spatial information. This invention addresses this by concatenating the results before passing them through the MLP, then passing them through a channel convolutional layer, essentially performing attention-based feature selection. This preserves useful information and reduces information loss. Finally, a fully connected (FC) layer maps the dimension to the channel dimension of the feature map.

[0265] The specific process is as follows: First, given the feature map F∈R C×H×W Using global average pooling and global max pooling, the spatial information is compressed and the dimensionality is eliminated to obtain: and The formula is as follows

[0266]

[0267]

[0268] Where MaxPool represents global average pooling and AvgPool represents global max pooling.

[0269] Then concatenate the two results to get F. concat =R 2C×1×1 As shown in the following formula

[0270]

[0271] Where concat represents the concatenation operation by channel.

[0272] Since the concatenated feature map has 2C channels, a single channel convolutional layer Con1x1 is needed to change the channel dimension to C, as shown in the following formula.

[0273] F′=Conv(F concat )

[0274] Where Conv represents a 1x1 channel convolution operation.

[0275] Finally, F′ is forwarded to an MLP (Multilayer Perceptron) network to compress the dimensionality, which is equivalent to using two fully connected (FC) layers to generate the channel attention feature map F. r ∈R C×1×1 Finally, the feature map is activated by the Sigmoid function. After the activation function, each weight value of the feature map will be in the range of (0,1).

[0276] F r =σ(MLP(F′))

[0277] Where σ represents the Sigmoid activation function.

[0278] The final formula for calculating the channel attention weights is as follows:

[0279] F r (F)-σ(MLP(Conv(Concat(Avg(F),Max(F))))

[0280] Where F r (F)=∈R C×1×1 Represents the channel weight matrix

[0281] After obtaining the attention weights for the channel features, the weights are multiplied by the original feature map to obtain the attention map. Finally, a residual connection is used to superimpose the attention map onto the original feature map, generating the final feature map. The formula is as follows:

[0282] F c =F·F r (F)+F

[0283] Where F c F is the resulting feature map, and F is the original feature map. r (F) represents the channel attention weight.

[0284] Local feature enhancement module, such as Figure 9 As shown, after the multi-scale feature extraction module, the feature dimensions of the three branches are high and the features are redundant. Therefore, the feature aggregation module is needed to further filter and enhance the important features to reduce feature redundancy and computation.

[0285] This module consists of channel convolution, pooling, unrolling, and fully connected (FC) layers. First, pooling reduces spatial resolution to increase the receptive field of features. Then, all channel features of each pixel are directly unrolled, followed by channel convolution and FC layers. This is equivalent to image feature-based attention, preserving important feature information from the feature map and removing redundant features to improve classification accuracy. The entire process can be represented by the following formula.

[0286] F=FC(Conv(Flattern(Pooling(x))))

[0287] Where x represents the input feature map, Pooling represents average pooling, Conv represents a channel convolution with a kernel size of 1, FC represents a fully connected layer, and Flattern represents expanding the features to eliminate spatial dimensions.

[0288] According to an embodiment of the present invention, the adaptive decision fusion module includes:

[0289] The adaptive decision fusion module uses an adaptive entropy weighted decision fusion method to assign different fusion weights to the softmax outputs of different networks.

[0290] It should be noted that during the testing phase, since there are multiple image patches for the final decision fusion, a feature fusion method is needed to fuse the results, such as... Figure 10 As shown, each image patch has its own softmax probability matrix, which describes the probability of each class as perceived by the model. The higher the probability of a certain class, the more likely the model thinks the image belongs to that class. However, it is also possible that two classes are very similar, in which case the model may consider the probability of both classes to be high. Therefore, the decision result is not very reliable in this case, so it should be given a lower weight to make its confidence level lower.

[0291] The maximum output probability decision fusion method is the simplest decision fusion method. It determines the final class of the input image by calculating the maximum probability of the outputs of different sub-networks. If the maximum output probability of each classifier falls within the same class, the decision result remains unchanged. However, as the probabilities change, the likelihood of the decision varies significantly. This method does not consider the weights assigned to each classifier during decision fusion; however, the fusion weight of each classifier determines the classification performance. Different sub-networks extract different features, and the samples have a certain degree of randomness, so a suitable decision method needs to be developed to obtain optimal classification performance.

[0292] An adaptive entropy-weighted decision fusion method is used to assign different fusion weights to the softmax outputs of different networks, further improving the classification accuracy of the algorithm while ensuring better model stability and robustness. This invention involves multiple image patches participating in the decision-making process; therefore, the fusion decision is obtained by adaptively entropy-weighted decision fusion of the probability values ​​from multiple image outputs to arrive at the final classification result. The specific adaptive entropy-weighted decision fusion algorithm is as follows.

[0293] When multiple softmax functions are input in parallel, the probability output matrix can be obtained as follows:

[0294]

[0295] Where m is the number of image patches, n is the number of categories, and p ij (x) represents the probability of the i-th image patch for class j. The column index of the highest probability in each row is the predicted class of the sample by the Softmax function for each image. The characteristics of the probability values ​​of the input sample belonging to each class will affect the classification accuracy. Specifically, the smaller the difference between the probability values, the greater the uncertainty of the classification; if the largest probability value is more different from the other probability values, the classification result is more reliable. Therefore, this invention introduces the information entropy H as shown in equation (2). i (x) is used to characterize the uncertainty of the input sample being classified by the Softmax function of the i-th sub-network.

[0296]

[0297] Where, p ij (x) represents the probability that the th subnetwork's Softmax function classifies the input sample as belonging to class . If the information entropy of a subnetwork's Softmax function is higher, the classification uncertainty is higher, and the Softmax function's ability to classify the input sample is worse. Therefore, the fusion weights of the Softmax function for the input sample are smaller, and vice versa. Thus, the adaptive fusion weights calculation formula for the Softmax function is:

[0298]

[0299] Therefore, the matrix is

[0300]

[0301] The label of the maximum value of the weighted sum of P(x) is the result of decision fusion, as shown in the formula.

[0302]

[0303] The adaptive entropy-weighted decision fusion algorithm fully considers the softmax function of different image patches and adaptively assigns more reasonable fusion weights to different input images, reducing the weight of image patches with high softmax matrix entropy.

[0304] Increase the weights of image patches with low softmax matrix entropy to improve classification accuracy.

[0305] According to an embodiment of the present invention, before sending the current animal fur image data to a preset first detection model, the method further includes:

[0306] Obtain dataset information;

[0307] The dataset information is augmented to obtain augmented dataset information;

[0308] The augmented dataset information is used for training according to preset rules;

[0309] The preset first detection model is obtained.

[0310] It should be noted that the loss function of this model uses the multi-class cross-entropy loss from the training phase to optimize the network.

[0311]

[0312] Where N represents the number of samples, i represents the sample index, c represents the class, M represents the number of classes, and p ic y represents the predicted probability that the observed sample belongs to class c. ic It is a sign function; it takes the value 1 if the true class of the sample is equal to c, and 0 otherwise.

[0313] like Figure 11 The dataset consists primarily of 24 categories of animal fur: mink, kangaroo, sheep, fox, weasel, sable, cow, American sable, sable, beaver, marmot, horse, chinchilla, stone marten, raccoon, fishing mink, raccoon, otter, coypu, gray squirrel, muskrat, raccoon dog, squirrel, and rabbit. The unsegmented dataset of 24 animal fur categories shows strong similarity in the fur of otters, raccoons, squirrels, and rabbits, while raccoon dog and horsehair also exhibit high similarity. Therefore, this dataset meets the characteristics of fine-grained image datasets, with large intra-class differences and small inter-class differences. Furthermore, key features are hidden in local image features (such as whether the fur edges have barbs, or whether the central part of the fur is distinctive). This dataset can be used for subsequent experiments. However, because the dataset was captured using a microscope, some parts contain significant noise, resulting in a low signal-to-noise ratio. This portion of the data may be discarded in subsequent processing.

[0314] Since the initial dataset contained only about 100 images per class with a resolution of 768*576, image segmentation was used to further expand the dataset and improve recognition performance. This involved dividing each image into smaller segments and removing images containing only the background. The final dataset consisted of 1500 images per class, for a total of 24 classes.

[0315] Because the background of images under a microscope is related to lighting conditions, it is difficult to guarantee that all images have the same background. Since images are segmented, the background in some images may occupy the majority of the image information. Therefore, the intensity of the background color also becomes a factor that interferes with image classification accuracy. Because the original microscopic images of animal fur contain multiple hairs, and the images of hair in different parts of the body vary greatly, with some images having many blank areas, this invention slices the dataset, dividing the animal fur images into 3×3 sub-images. This not only enhances the features in the images and improves image quality, but also expands the dataset, which is beneficial for improving classification accuracy.

[0316] Edge pixel detection is used to examine pixels at the edges of an image. Since fur, which exists only in a small area, is often excessively noisy and can interfere with the classification of other image categories, we remove it. We detect pixel values ​​in the outer 20% region of the image starting from the four borders, and each new pixel value is denoted as K. i Record K i The number, if K i If the number of occurrences is less than the threshold, the image is removed. The specific expression is as follows:

[0317]

[0318] After sub-image segmentation, the detailed features of the image are more clearly displayed. At the same time, the differences between different fur types are also clearer, and the fur features of different categories can be more easily distinguished.

[0319] Processed image as follows Figure 12 As shown, the advantages of this processing are that it expands the dataset and enhances the class features of the dataset. Using the processed image dataset can amplify image detail information in a single image, which is beneficial for subsequent network feature extraction.

[0320] According to an embodiment of the present invention, data augmentation processing is performed on the dataset information to obtain augmented dataset information, including:

[0321] Data augmentation processing of the dataset information includes rotation, flipping, scaling, translation, random cropping, changing brightness, changing saturation, and changing hue.

[0322] It should be noted that, to avoid underfitting or overfitting due to image sample quality and dataset diversity, preprocessing methods such as rotation, flipping, scaling, translation, random cropping, brightness adjustment, saturation adjustment, and hue adjustment are employed. For flipping, each training image is flipped horizontally or vertically with a 50% probability, allowing the network to fully observe image changes during training and enhancing its robustness. For rotation, a small 10-degree rotation is performed around the image origin. In the translation method, the image is slightly shifted vertically and horizontally to enhance the network's robustness. In random cropping, a portion of the image is cropped and enlarged to a size of 224*224, allowing for the extraction of finer-grained features, which is more suitable for the microscopic images studied in this invention. In the brightness, saturation, and hue adjustment methods, these three attributes are randomly changed within a range of 1 to 1.35. Within this range, the image's visibility remains unchanged, further enhancing the model's robustness.

[0323] According to an embodiment of the present invention, it further includes:

[0324] The dataset includes a training set, a validation set, and a test set.

[0325] It should be noted that the experiment used a self-made dataset and a backbone network pre-trained on ImageNet. The training and validation sets were used for network training, and the test set was used for network testing, with a ratio of 8:1:1. The operating environment was Python 3.8 and PyTorch 1.7. The image enhancement methods used included random cropping, random changes to image brightness, saturation, and hue (ranging from 1 to 1.35 of the original image), horizontal flipping, and vertical flipping. Stochastic gradient descent (SGD) with momentum was chosen as the optimizer, employing a cosine roll-off learning rate decay strategy with an initial learning rate of 0.01. The training consisted of 50 epochs with a batch size of 32 and input images of 224*224 pixels. The server was running Ubuntu 18.04, with an i9 10900K CPU, a 3090 graphics card with 24GB of VRAM, and 128GB of RAM.

[0326] According to an embodiment of the present invention, it further includes:

[0327] Ablation experiment.

[0328] It should be noted that, firstly, this invention uses image segmentation to expand the dataset, increasing the original dataset of 100 images per class to 1500 images per class. The effect before and after using this method is shown in the figure.

[0329] Method Acc ResNet50 0.747 ResNet50+Pieces 0.802

[0330] As can be seen, manipulating the dataset can improve the recognition results to some extent, which can be considered a form of image enhancement. All subsequent methods will use this method and will not be elaborated further.

[0331] Method Acc ResNet50 0.802 ResNet50+SENet 0.832 ResNet50+CBAM 0.835 ResNet50+ Attention Module 0.844

[0332] The table compares existing attention modules and shows that the attention module of this invention performs better than both SENet and CBAM.

[0333] Method Acc ResNet50 0.802 +Multi-scale spatial attention module 0.837 +Multi-scale attention module 0.844 +Adaptive Fusion Decision Module 0.863

[0334] As shown in the table, the accuracy of the original ResNet50 network is only 0.802. After adding the multi-scale attention module, the model can extract multi-scale features including texture and semantic features with an accuracy of 0.855. After adding the feature aggregation module, the accuracy is 0.863. After adding the adaptive fusion decision module, the results of multiple image patches are added with adaptive weights and make joint decisions, and the accuracy can reach 0.889. It can be seen that the improved method has good reliability in recognition performance.

[0335] According to an embodiment of the present invention, it further includes:

[0336] Comparative experiment.

[0337] It should be noted that the parameter comparison involves comparing the multi-scale attention module with FPN and Unet structures, and also comparing the results of different parameters in classic network models such as VGG (e.g., hyperparameter sizes like patch size in VIT), loss curves, accuracy curves, F1 score curves, confusion matrices, Gradcam plots (visualization results), including those after adding various modules, as well as comparisons of class classification performance with other algorithms, such as... Figure 13 As shown.

[0338] Algorithm Acc ResNet50 0.802 ResNet50+CBAM 0.835 MobileNetV3-Small 0.738 EfficientNetV2-S 0.863 MobileViT-X 0.773 Vision Transformer 0.852 ResNet101 0.854 Ours 0.863

[0339] As can be seen, the original ResNet50 network has an accuracy of only 0.802. Adding the CBAM attention module to the original ResNet50 network improves the accuracy to 0.835. The EfficientNetV2-S model, proposed in 2021, is a network with faster training speed, fewer parameters, and a progressive learning strategy [citation needed], achieving an accuracy of 0.837. MobileViT-X, also proposed in 2021, uses MobileNetV3 as its backbone network and a hybrid architecture of CNN and Transformer, achieving an accuracy of 0.773. However, it has a very small number of parameters, resulting in fast training and inference speeds. ResNet101 is an upgraded version of ResNet50, increasing network depth and parameters, achieving an accuracy of 0.854. The algorithm of this invention achieves an accuracy of 0.863, outperforming other methods and demonstrating its effectiveness.

[0340] A third aspect of the present invention provides a computer-readable storage medium comprising a multi-scale attention fur image classification method program, wherein when the multi-scale attention fur image classification method program is executed by a processor, it implements the steps of the multi-scale attention fur image classification method as described in any of the preceding claims.

[0341] This invention discloses a multi-scale attention-based method, system, and storage medium for fur image classification. The proposed method utilizes channel and spatial attention, a multi-scale feature extraction network, and an adaptive decision fusion method to accurately identify and classify animal microscopic fur images. First, channel attention combines cross-correlation between channels with a feedforward convolutional neural network branch, while spatial attention combines cross-correlation between spatial locations with a feedforward convolutional neural network branch. Then, a bottom-up propagation branch is used to pass low-level multi-channel semantic features to higher levels. Next, a top-down propagation branch is used to fuse high-resolution spatial features from higher levels back to lower levels. Features from the three different branches are then fused through a feature aggregation network. Finally, an entropy-based adaptive decision fusion method is used to obtain the final classification result.

[0342] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0343] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0344] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0345] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0346] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A multi-scale attention-based fur image classification method, characterized in that, include: Get the current animal fur image data; The current animal fur image data is sent to a preset first detection model; Obtain the classification results information; The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module. The multi-scale feature extraction module includes a bottom-up propagation branch and a top-down propagation branch. First, it fuses high-level semantic features into the low-level layer through element-wise addition to obtain the low-level fusion result. Then, it fuses the low-level fusion result into the high-level layer through channel concatenation. An improved channel attention module is embedded in the bottom-up propagation branch, and an improved spatial attention module is embedded in the top-down propagation branch. The improved channel attention module is an improvement on the channel attention module in CBAM. Specifically, it performs global average pooling and global max pooling on the input feature map, concatenates the two pooling results by channel, adjusts the channel dimension through a 1×1 convolutional layer, inputs it into an MLP network to generate channel attention weights, and finally fuses the attention weights with the original feature map through residual connections. The improved spatial attention module is an improvement on the spatial attention module in CBAM. Specifically, it performs global average pooling and global max pooling on the input feature map, concatenates the two pooling results by channel, adjusts the channel dimension through a 1×1 convolutional layer, and then inputs it into an MLP network consisting of two fully connected layers to generate spatial attention weights. Finally, it fuses the attention weights with the original feature map through residual connections. The adaptive decision fusion module employs an adaptive entropy weighted decision fusion method. It calculates the information entropy of the softmax outputs corresponding to multiple image blocks after the input image is sliced. Based on the information entropy, it adaptively assigns fusion weights to each softmax output, reducing the weight of high-entropy image blocks and increasing the weight of low-entropy image blocks. The final classification result is obtained after weighted summation.

2. The multi-scale attention-based fur image classification method according to claim 1, characterized in that, The multi-scale feature extraction module specifically comprises: The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result. Integrate the results from the lower level into the higher level; Spatial attention module and channel attention module were added to the two branches.

3. The multi-scale attention-based fur image classification method according to claim 2, characterized in that, The spatial attention module includes: Obtain the spatial attention module in CBAM; The spatial attention module in the CBAM is improved according to preset rules; The spatial attention module is obtained.

4. The multi-scale attention-based fur image classification method according to claim 2, characterized in that, The channel attention module includes: Obtain the channel attention module in CBAM; The channel attention module in the CBAM is improved according to preset rules; The channel attention module is obtained.

5. The multi-scale attention-based fur image classification method according to claim 1, characterized in that... The adaptive decision fusion module includes: The adaptive decision fusion module uses an adaptive entropy weighted decision fusion method to assign different fusion weights to the softmax outputs of different networks.

6. The multi-scale attention-based fur image classification method according to claim 1, characterized in that, Before sending the current animal fur image data to the preset first detection model, the process also includes: Obtain dataset information; The dataset information is augmented to obtain augmented dataset information; The augmented dataset information is used for training according to preset rules; The preset first detection model is obtained.

7. A multi-scale attention-based fur image classification system, characterized in that, The system includes a memory and a processor. The memory contains a program for a multi-scale attention-based fur image classification method. When executed by the processor, the multi-scale attention-based fur image classification method program performs the following steps: Get the current animal fur image data; The current animal fur image data is sent to a preset first detection model; Obtain the classification results information; The preset first detection model includes a feature extraction basic module, a multi-scale feature extraction module, a feature aggregation module, and an adaptive decision fusion module. The multi-scale feature extraction module includes a bottom-up propagation branch and a top-down propagation branch. First, it fuses high-level semantic features into the low-level layer through element-wise addition to obtain the low-level fusion result. Then, it fuses the low-level fusion result into the high-level layer through channel concatenation. An improved channel attention module is embedded in the bottom-up propagation branch, and an improved spatial attention module is embedded in the top-down propagation branch. The improved channel attention module is an improvement on the channel attention module in CBAM. Specifically, it performs global average pooling and global max pooling on the input feature map, concatenates the two pooling results by channel, adjusts the channel dimension through a 1×1 convolutional layer, inputs it into an MLP network to generate channel attention weights, and finally fuses the attention weights with the original feature map through residual connections. The improved spatial attention module is an improvement on the spatial attention module in CBAM. Specifically, it performs global average pooling and global max pooling on the input feature map, concatenates the two pooling results by channel, adjusts the channel dimension through a 1×1 convolutional layer, and then inputs it into an MLP network consisting of two fully connected layers to generate spatial attention weights. Finally, it fuses the attention weights with the original feature map through residual connections. The adaptive decision fusion module employs an adaptive entropy weighted decision fusion method. It calculates the information entropy of the softmax outputs corresponding to multiple image blocks after the input image is sliced. Based on the information entropy, it adaptively assigns fusion weights to each softmax output, reducing the weight of high-entropy image blocks and increasing the weight of low-entropy image blocks. The final classification result is obtained after weighted summation.

8. A multi-scale attention-based fur image classification system according to claim 7, characterized in that, The multi-scale feature extraction module specifically comprises: The multi-scale feature extraction module first performs element-wise summation to fuse high-level semantic features into the low-level layer, thus obtaining the low-level result. Integrate the results from the lower level into the higher level; Spatial attention module and channel attention module were added to the two branches.

9. A multi-scale attention-based fur image classification system according to claim 8, characterized in that, The spatial attention module includes: Obtain the spatial attention module in CBAM; The spatial attention module in the CBAM is improved according to preset rules; The spatial attention module is obtained.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a multi-scale attention fur image classification method program, which, when executed by a processor, implements the steps of the multi-scale attention fur image classification method as described in any one of claims 1 to 6.

Images