Long-tail image data classification method and device, electronic equipment and storage medium

Abstract

The invention provides a long-tail image data classification method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target long-tail image, processing the target long-tail image through employing a long-tail image classification model, and obtaining a classification result of the target long-tail image; the long-tail image classification model is trained by the following steps: acquiring a long-tail data set, and performing feature extraction on the long-tail data set by adopting a ResNet backbone network to obtain multi-level hierarchical features; performing enhancement processing on the multi-level hierarchical features by adopting an adaptive context to obtain enhanced multi-level hierarchical features; a ReLU activation function is adopted to classify the enhanced multi-level hierarchical features based on linear growth and a nonlinear gating mechanism, and a classification result of the long-tail data set is obtained; and completing the training of a long-tail image classification model according to the classification result of the long-tail data set. The method has the beneficial effect that the accuracy of long-tail picture data classification is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence and image recognition technology, and in particular to a long-tail image data classification method, device, electronic equipment and storage medium. Background Art

[0002] In real-world scenarios, naturally collected data often presents a long-tail distribution. Long-tail distribution refers to the data imbalance phenomenon in which most categories have relatively few samples, while a few categories have significantly more samples. For example, there are multiple faces in an image, occupying most of the image, while non-face areas occupy a smaller area. Long-tail distribution is widely present in various practical applications such as face recognition, species classification, and medical image analysis. For example, in medical images, common diseases such as retinal abnormalities caused by pneumonia and diabetes may be relatively easy to detect and identify; however, rare types of cancer or unusual tumor morphologies require special attention.

[0003] Therefore, based on the long-tail distribution data mentioned above, the usual classification method will lead to missing important information and inaccurate data classification.

[0004] Traditional long-tail classification methods include random oversampling (ROS), random undersampling (RUS) and category reweighting techniques. They also include improving data-level methods, optimizing model architectures and implementing feature balancing strategies, and solving the problem of imbalanced class distribution through new sampling techniques, resampling methods and data augmentation. In addition, feature selection and representation learning are emphasized to enhance the discriminative ability of the model, especially for minority category samples. For example, long-tail recognition is handled through two network branches: traditional learning and rebalancing.

[0005] However, for image recognition, the above-mentioned existing technologies mainly consider the problem of category imbalance, while ignoring the gap between categories and fine-grained recognition, thus resulting in inaccurate image classification. Summary of the invention

[0006] The main purpose of the embodiments of the present invention is to provide a long-tail image data classification method, device, electronic device and storage medium, which improves the accuracy of long-tail image data classification.

[0007] One aspect of the present invention provides a long-tail image data classification method, comprising:

[0008] Acquire a target long-tail image, and process the target long-tail image using a long-tail image classification model to obtain a classification result of the target long-tail image;

[0009] The long-tail image classification model is trained by the following steps:

[0010] Obtain a long-tail data set, and use a ResNet backbone network to perform feature extraction on the long-tail data set to obtain multi-level hierarchical features;

[0011] Adopting adaptive context to enhance the multi-level hierarchical features to obtain enhanced multi-level hierarchical features;

[0012] The enhanced multi-level hierarchical features are classified using a ReLU activation function based on a linear growth and nonlinear gating mechanism to obtain a classification result of the long-tail data set;

[0013] According to the classification result of the long-tail data set, the training of the long-tail image classification model is completed.

[0014] According to the long-tail image data classification method, a long-tail data set is obtained, and a ResNet backbone network is used to extract features from the long-tail data set to obtain multi-level hierarchical features, including:

[0015] An input image is extracted from the long-tailed dataset, and according to the dataset type of the long-tailed dataset, a corresponding ResNet backbone network is selected to perform feature extraction using a dynamic pooling layer, wherein the dynamic pooling layer dynamically adjusts the pooling operation parameters by dynamically adjusting the pooling window and according to the feature distribution of the input image.

[0016] According to the long-tail image data classification method, the multi-level hierarchical features are enhanced using adaptive context to obtain enhanced multi-level hierarchical features, including:

[0017] The low-level features, middle-level features and high-level features of the multi-level hierarchical features are obtained, and the features of different levels are enhanced by adaptive context through different numbers of channels to obtain enhanced multi-level hierarchical features. The enhanced multi-level hierarchical features are fused by element-by-element addition. The formula for the enhanced multi-level hierarchical features is:

[0018]

[0019] in, To enhance the multi-level features, f i (i=1; 2; 3) is a multi-level hierarchical feature and i represents the level, BN is normalization processing, Conv 3×3 It is 3×3 convolution processing, and ACM is adaptive context processing.

[0020] According to the long-tail image data classification method, the features at different levels are enhanced by adaptive context through different numbers of channels to obtain enhanced multi-level features, and the method also includes:

[0021] The unenhanced intermediate layer features are fused with the enhanced high-level features using cross-layer connections. The fusion process is:

[0022] f m =Maxpooling(Conv 1×1 (f2))

[0023] Among them, f m is the intermediate layer feature after fusion, Maxpooling is the maximum pooling process, Conv 1×1 It is a 1×1 convolution process and is an intermediate layer feature without enhancement.

[0024] According to the long-tail image data classification method, the multi-level hierarchical features are enhanced using adaptive context to obtain enhanced multi-level hierarchical features, and further comprising:

[0025] Multi-scale pooling and attention mechanisms are used to process feature maps at different levels, including average pooling operations on input feature maps of different scales to aggregate contextual information to varying degrees. The process is as follows:

[0026] x pooled =Avgpool(x,scale)

[0027] Where Avgpool is the average pooling process, x pooled It is the feature map obtained after average pooling, where scale = [1, 2, 4].

[0028] According to the long-tail image data classification method, the multi-level hierarchical features are enhanced by element-by-element addition fusion, including:

[0029] The enhanced features of the low-level features, the enhanced features of the middle-level features, the enhanced features of the high-level features, and the fused middle-level features are added element by element, and the fused features of different levels are sent to the classifier. The addition process formula is:

[0030]

[0031] f′ is the fused multi-level feature, They are the enhanced features of low-level features, the enhanced features of middle-level features, and the enhanced features of high-level features, respectively. m It is the intermediate layer feature after fusion.

[0032] According to the long-tail image data classification method, the enhanced multi-level hierarchical features are classified based on linear growth and nonlinear gating mechanisms using a ReLU activation function to obtain the classification results of the long-tail data set, including:

[0033] Next, after pooling the feature maps of each layer, convolution operation is performed on the feature maps, and then nonlinear transformation is performed using the ReLU activation function, and the Sigmoid activation function is used to map the output to the range of 0 to 1 to obtain the attention map. The processing process is as follows:

[0034] Map = Sigmoid(Conv 1×1 (ReLU(Conv 1×1 (X pooled ))))

[0035] Among them, Map represents the attention map, Sigmoid represents the input mapping process of the Sigmoid activation function, ReLU represents the nonlinear transformation process of the ReLU activation function, and Conv 1×1 represents 1×1 convolution processing, X pooled is the feature map of average pooling;

[0036] Perform bilinear interpolation on the generated attention map, including adjusting the attention map to the same size as the original input feature map, and concatenating the attention maps generated on all levels along the channel dimension to obtain multi-scale contextual attention features;

[0037] The ReLU activation function uses the LT-ReLU activation function, which uses a linear growth mechanism and a nonlinear gating mechanism. The linear growth mechanism is used to show linear growth characteristics in dense data areas, and the nonlinear gating mechanism is used to introduce nonlinear elements in sparse data areas. The formula is:

[0038] f(x)=max(x,0)+(1-max(x,0))·σ(x-λ)

[0039] Among them, λ is an optimizable adaptive threshold parameter, max(x,0) is a linear growth, and (1-x)·sigmoid(x-λ) is a nonlinear gating.

[0040] And, use the tanh function to limit the overall output range to [-1, 1], the formula is

[0041]

[0042] Where τ is the scaling factor.

[0043] Another aspect of an embodiment of the present invention provides a long-tail image data classification device, comprising:

[0044] The first module is used to obtain a target long-tail image, and process the target long-tail image using a long-tail image classification model to obtain a classification result of the target long-tail image;

[0045] The long-tail image classification model is trained using the following modules:

[0046] The second module is used to obtain a long-tail data set, and use a ResNet backbone network to extract features from the long-tail data set to obtain multi-level hierarchical features;

[0047] The third module is used to enhance the multi-level hierarchical features by using adaptive context to obtain enhanced multi-level hierarchical features;

[0048] The fourth module is used to classify the enhanced multi-level hierarchical features based on linear growth and nonlinear gating mechanisms using a ReLU activation function to obtain a classification result of the long-tail data set;

[0049] The fifth module is used to complete the training of the long-tail image classification model according to the classification results of the long-tail data set.

[0050] Another aspect of an embodiment of the present invention provides an electronic device, including a processor and a memory;

[0051] The memory is used to store programs;

[0052] The processor executes the program to implement the method described above.

[0053] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method described above.

[0054] The beneficial effects of the present invention are as follows: by adaptively and dynamically adjusting the receptive field size, and dynamically adjusting the size and span of the pooling window to meet the needs of feature distribution of different categories, the feature extraction efficiency of long-tail data is improved; a multi-scale pool and attention mechanism are used to process feature maps of different levels, and the attention mechanism is used to integrate features into feature representation, thereby enhancing the model's understanding and representation capabilities of input data; a new ReLU function is used to cope with the output of dense data and sparse data distribution based on a linear growth mechanism and a nonlinear gating mechanism, respectively, thereby improving sensitivity to data changes; and the accuracy of long-tail image data classification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] Figure 1It is a schematic diagram of the long-tail data classification process of an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of an ACM module according to an embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of a long-tail data classification model according to an embodiment of the present invention.

[0059] Figure 4 Schematic diagram of a long-tail data classification device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. In the subsequent description, the use of suffixes such as "module", "component" or "unit" used to represent elements is only for the purpose of facilitating the description of the present invention, and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner. "First", "second" and the like are only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the continuous numbering of the method steps is for the convenience of review and understanding. In combination with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0061] In some embodiments, reference Figure 1 ,in Figure 1 : is a schematic diagram of the long-tail data classification process, which includes but is not limited to steps S100 to S500:

[0062] S100, obtaining a target long-tail image, and processing the target long-tail image using a long-tail image classification model to obtain a classification result of the target long-tail image;

[0063] It can be understood that, after training and learning, the embodiment of the present invention obtains a long-tail image classification model.

[0064] S200, obtain a long-tail data set, use the ResNet backbone network to extract features from the long-tail data set, and obtain multi-level hierarchical features.

[0065] In some embodiments, an input image is extracted from a long-tail dataset, and based on the dataset type of the long-tail dataset, a corresponding ResNet backbone network is selected to perform feature extraction using a dynamic pooling layer, wherein the dynamic pooling layer dynamically adjusts the pooling window and dynamically adjusts the pooling operation parameters based on the feature distribution of the input image.

[0066] In some embodiments, for an RGB input image I of size H×W, the embodiment of the present invention uses ResNet-32 as the backbone network on the CIFAR-100-LT dataset and uses ResNet-50 as the backbone network on the Imagenet-LT dataset to extract multi-level features from the input image, denoted as f i (i=1; 2; 3).

[0067] S300, performing enhancement processing on the multi-level hierarchical features by using adaptive context to obtain enhanced multi-level hierarchical features.

[0068] In some embodiments, low-level features, middle-level features, and high-level features of multi-level hierarchical features are obtained, and different levels of features are enhanced by adaptive context through different numbers of channels to obtain enhanced multi-level hierarchical features, and the enhanced multi-level hierarchical features are fused by element-by-element addition. The formula for obtaining the enhanced multi-level hierarchical features is:

[0069]

[0070] in, To enhance the multi-level features, f i (i=1; 2; 3) is a multi-level hierarchical feature and i represents the level, BN is normalization processing, Conv 3×3 It is 3×3 convolution processing, and ACM is adaptive context processing.

[0071] In some embodiments, reference Figure 2 The ACM module schematic diagram shown in the figure shows that the ACM in the embodiment of the present invention is an adaptive context module.

[0072] Feature enhancement plays a vital role in improving the model's ability to represent input data or extract specific features. The main methods include data enhancement, feature expansion, feature selection, and feature reconstruction. Therefore, the embodiment of the present invention proposes a feature enhancement module that integrates an adaptive receptive field (ARF) and a multi-scale contextual attention module (MS-CAM). Figure 2 As shown, after the input image I is preliminarily processed and low-, medium-, and high-dimensional features are extracted, the feature maps are input into the adaptive context module (ACM) accordingly.

[0073] The adaptive receptive field module helps to dynamically adjust the receptive field size of the network. By introducing a dynamic pooling layer, the embodiment of the present invention can dynamically adjust the size and span of the pooling window to meet the needs of feature distribution of different categories. Introducing learning parameters in this module allows the network to dynamically adjust the pooling operation parameters according to the input data features, thereby maximizing the efficiency of feature extraction.

[0074] At the same time, the multi-scale contextual attention mechanism effectively integrates information of different scales by generating feature maps of different scales. In deep learning, contextual features carry rich semantic information. In many visual tasks, objects or scenes may have features of different scales, which limits the ability of traditional neural networks to process such multi-scale information.

[0075] In some embodiments, in MS-CAM, the embodiments of the present invention use multi-scale pooling and attention mechanisms to process feature maps at different levels. First, the embodiments of the present invention set scale = [1, 2, 4], perform average pooling operations on input feature maps of different scales, and aggregate context information to different degrees. This process can be described as follows:

[0076] x pooled =Avgpool(x,scale)

[0077] Next, after pooling the feature maps at each scale, the feature maps are convolved and then nonlinearly transformed using the ReLU activation function. The feature maps are then further processed using a 1x1 convolutional layer to obtain the attention map. Here, the Sigmoid activation function is used to map the output to a range of 0 to 1, indicating the importance of each pixel. The process is as follows:

[0078] Map = Sigmoid(Conv 1×1 (ReLU(Conv 1×1 (X pooled ))))

[0079] Finally, in order to maintain the spatial consistency between the attention map and the original feature map, the embodiment of the present invention performs bilinear interpolation on the generated attention map and adjusts it to the same size as the original input feature map. Subsequently, the attention maps generated at all scales are concatenated along the channel dimension to obtain the final multi-scale contextual attention features. Experimental results show that the module of the embodiment of the present invention can capture contextual information of different scales and integrate it into the feature representation using the attention mechanism, thereby enhancing the model's understanding and representation capabilities of the input data.

[0080] S400, uses the ReLU activation function to classify the enhanced multi-level hierarchical features based on linear growth and nonlinear gating mechanism to obtain the classification results of the long-tail dataset.

[0081] Understandably, the traditional ReLU activation function has limitations to a certain extent. When the input is less than zero, the gradient becomes zero, which may cause some neurons to be inactive indefinitely and unable to recover. In addition, this situation may also lead to unstable gradient changes, especially near negative values, causing the "gradient explosion" problem. Therefore, researchers have proposed various improved ReLU functions, such as PReLU, LeakyReLU, etc.

[0082] Considering that long-tail data distribution has the characteristics of nonlinearity, long tail, gradually decreasing density and a certain degree of sparsity, the embodiment of the present invention proposes a new activation function, which combines the neuron activation process observed in biological neurons. Therefore, when designing an activation function, it is necessary to consider the introduction of nonlinear elements and how to deal with sparse data.

[0083] In biological neural systems, the activation process of neurons responding to input can be simplified into two stages: a linear response stage and a nonlinear saturation stage. Therefore, the activation function proposed in the embodiment of the present invention simulates these two stages, wherein the linear part simulates the linear response of biological neurons, and the sigmoid function simulates the response of the saturated nonlinear stage.

[0084] Taking these factors into consideration, the activation function LT-ReLU of the embodiment of the present invention combines linear growth and nonlinear gating mechanisms, which enables it to exhibit linear growth characteristics in dense data areas, while introducing nonlinear elements in sparse data areas, thereby better adapting to uneven data distribution.

[0085] f(x)=max(x,0)+(1-max(x,0))·σ(x-λ)

[0086] In this case, λ represents an optimizable adaptive threshold parameter. It is compared with the value of x to determine whether a data point belongs to the dense head region or the sparse tail region. During training, the model dynamically selects the value of the threshold λ based on the distribution characteristics of the data to adapt to the characteristics of different data distributions.

[0087] In the dense head area of ​​the data, the embodiment of the present invention hopes that the output of the function is linearly related to the input, that is, the increase in the input value directly leads to the increase in the output value. In addition, this growth should be relatively stable and not affected by too many nonlinear factors. In the tail area of ​​the data where the data points are relatively sparse, the goal of the embodiment of the present invention is to make the function more sensitive to data changes. This is mainly achieved through two parts: the linear growth part x, and the nonlinear element (1-x)·sigmoid(x-λ) introduced by the nonlinear gating mechanism. The embodiment of the present invention uses the sigmoid function to map (x-λ) to the interval [0,1], thereby introducing a nonlinear transformation to adapt to the characteristics of the long-tail data distribution, and perform more complex transformations on the data when necessary.

[0088] Subsequently, in order to alleviate the imbalance of long-tail data and adapt to the differences in input ranges of different categories, the embodiment of the present invention uses the tanh function to limit the overall output range to [-1, 1]. The specific performance is as follows:

[0089]

[0090] In this case, τ is a scaling factor, and larger values ​​of τ make the activation function smoother, helping to mitigate sudden changes in gradients and keep it stable during training.

[0091] S500, completing the training of the long-tail image classification model according to the classification results of the long-tail data set.

[0092] In some embodiments, reference Figure 3 The overall framework diagram of the multi-branch network TVNet shown in the figure, its low-level features provide detailed information of the image, such as contours, edges and textures; high-level features provide rich semantic information, capturing the abstract concepts and category characteristics of objects.

[0093] like Figure 3 As shown, the classification model proposed in the embodiment of the present invention adopts a multi-branch network to integrate low-level and high-level features, wherein the first layer, the second layer, and the third layer are used to process low-level features, intermediate features, and high-level features, respectively.

[0094] Specifically, for an RGB input image I of size H×W, the embodiment of the present invention uses ResNet-32 as the main network on the CIFAR-100-LT dataset and uses ResNet-50 as the main network on the Imagenet-LT dataset to extract multi-level features from the input image, denoted as f i (i=1; 2; 3). Then, the embodiment of the present invention combines the features f at different levels i(i=1; 2; 3) Input the adaptive context module (ACM) with different number of channels for feature enhancement, and use element-by-element addition to fuse the enhanced multi-level features. The enhanced features of each layer f i Represented as f i A The process can be described as follows:

[0095]

[0096] In addition, the embodiment of the present invention also introduces cross-layer connections to promote information flow, that is, the original intermediate layer features that have not been enhanced are processed to promote their fusion with the high-level features processed by ACM. The purpose of this is to ensure the effective interaction between local details and global semantic information during the feature fusion process, especially for those details that may be ignored or blurred in the high-level features.

[0097] The original intermediate layer features may contain more low-level details and local information, while the high-level features after feature enhancement may contain richer abstract semantic information. This process can be expressed as follows, where f m represents the processed intermediate layer features, and f2 represents the original intermediate layer features.

[0098] f m =Maxpooling(Conv 1×1 (f2))

[0099] Here, the embodiment of the present invention implements the fusion of the original intermediate layer features and the high-level features by applying the Conv1×1 convolution layer on the feature map, thereby promoting linear transformation and information fusion. At the same time, the embodiment of the present invention uses the MaxPooling operation to reduce the spatial dimension of the original feature map, thereby further extracting and abstracting features.

[0100] Finally, through the element-by-element addition operation, the fusion features at different levels are input into the classifier for classification. This process can be described as follows:

[0101]

[0102] f″=Avgpooling(Conv 1×1 (ReLU(f′))

[0103] Among them, f′ represents the fused multi-level features, and f″ represents the final feature output.

[0104] In some embodiments, the proposed TVNet architecture is validated on two widely used long-tail recognition benchmark datasets: CIFAR-100-LT and ImageNet-LT. CIFAR-100-LT and ImageNet-LT are artificially unbalanced datasets, in contrast to their balanced counterparts CIFAR-100 and ImageNet-2012. Throughout the experiment, the embodiments of the present invention strictly adhere to the official training and test splits for each dataset.

[0105] In some embodiments, the Top-1 accuracy of the entire data set is used as the main performance indicator. In addition, the embodiments of the present invention also list the accuracy of three different subsets: the head class (consisting of classes with more than 100 training samples), the middle class (including classes with 20 to 100 samples) and the tail class (consisting of classes with less than 20 samples).

[0106] In some embodiments, all experimental processes of the embodiments of the present invention are completely implemented using PyTorch. In the CIFAR-100-LT dataset, the embodiments of the present invention use ResNet-32 as the model architecture. According to the prescribed training method, the network was fully trained 200 times. For the ImageNet-LT dataset, the embodiments of the present invention use ResNet-50 as the base network. During training, the initial learning rate was 0.1 for 100 rounds, and the attenuation coefficient for the 60th and 80th rounds was 0.1. All experiments used stochastic gradient descent (SGD) with a momentum value of 0.9.

[0107] For the ReLU function proposed in the embodiment of the present invention, the embodiment of the present invention sets the threshold parameter λ to a small value so that most samples can pass through the sigmoid part while retaining a small part for the linear part. Considering the imbalance of the data set, the embodiment of the present invention sets it to a small negative value of -0.5. After passing through the sigmoid and linear stages, the embodiment of the present invention sets the parameter τ Initialized to a small positive value of 0.1 in order to compress the overall output range into a more limited interval.

[0108] The embodiment of the present invention compares TVNet and several commonly used long-tail recognition methods on two data sets.

[0109] Minority oversampling techniques: (1) No oversampling (vanilla); (2) Random Oversampling (ROS), adding minority samples during training to balance the class distribution; (3) Remix; (4) Background-rich Minority Oversampling (CMO); (5) Feature Space Augmentation (FSA).

[0110] Weighting schemes: (6) Label Distribution Aware Margin Loss (LDAM), which adjusts the minority class samples to push the margin towards the decision boundary; (7) Influence Balanced Loss (IB), which reweights the samples according to their influence; (8) Balanced Softmax, an unbiased extension of Softmax; (9) LADE, which separates the source label distribution from the model prediction.

[0111] Other recent methods: (10) Delayed Reweighting (DRW) and (11) Decoupling, a two-stage algorithm that rebalances classifiers during fine-tuning; (12) BBN, which employs additional network branches to address the class imbalance problem.

[0112] CIFAR-100-LT. The embodiments of the present invention are experimented on CIFAR-100-LT using different imbalance ratios: 10, 50, and 100. TVNet is applied to various methods to verify its effectiveness under different algorithms, namely, vanilla cross entropy loss, class balance loss (LDAM), and two-stage algorithm (DRW).

[0113] Table 1 shows the overall classification accuracy compared with the state-of-the-art methods. Combining the representative methods in the long-tail domain with TVNet improves the performance when the dataset imbalance ratio is 100, 50, and 10, respectively. In addition, the embodiment of the present invention also compares the performance improvement brought by combining TVNet with the long-tail recognition methods, as shown in Table 2. The results show that when used in combination with TVNet, the performance of all long-tail recognition methods is consistently improved. On the other hand, using ROS for simple class distribution balancing greatly degrades the performance.

[0114] Table 1. State-of-the-art comparison on the CIFAR-100-LT dataset. Classification accuracy on the CIFAR-100-LT dataset with different imbalance ratios using the ResNet-32 architecture.

[0115]

[0116] Table 2 Comparison of classification accuracy on the CIFAR-100-LT benchmark with ResNet-32 (imbalance ratio = 100).

[0117]

[0118] ImageNet-LT, where Table 3 lists the results of the method of the embodiment of the present invention and other long-tail recognition methods. Combining TVNet with baseline training using cross entropy (CE) loss, the performance improvement exceeds the recent baseline. However, compared with the Resnet-32 used in CIFAR-100-LT, thanks to the richer contextual information in the main categories of ImageNet-LT, the results show that the method of the embodiment of the present invention still has some improvements. In addition, Table 4 also shows that incorporating BS into TVNet can improve performance, indicating that the model of the embodiment of the present invention can be seamlessly integrated into modern state-of-the-art long-tail recognition methods.

[0119] Table 3. State-of-the-art ImageNet-LT comparison. Classification accuracy of ResNet-50 trained with state-of-the-art methods in 90 or 100 epochs.

[0120]

[0121] Table 4 Comparison of classification accuracy on ResNet-50 with ImageNet-LT benchmark.

[0122]

[0123] In order to verify the effectiveness of each key module, the embodiment of the present invention conducted a series of ablation experiments on the CIFAR-100-LT dataset.

[0124] Ablation of model structure. As shown in Table 5, the embodiment of the present invention conducted an ablation experiment on the CIFAR-100-LT dataset to verify the effectiveness of each module in TVNet. The multi-branch architecture, ARF, and MS-CAM were integrated into the ResNet-32 architecture respectively, and their performance was compared. The results show that in ResNet-32, integrating the multi-branch structure can significantly improve the performance under different imbalance ratios (100, 50, and 10). However, adding the ARF module or the MS-CAM module alone can only bring limited performance improvement, or even limit the performance. Therefore, based on the experimental results, the embodiment of the present invention concludes that integrating these two modules is crucial to fully tap the potential of feature extraction, and the fusion layer is the key to maximizing its value. The embodiment of the present invention takes the Top-1 accuracy of the entire dataset as the main performance indicator. In addition, the embodiment of the present invention also lists the accuracy of three different subsets: the head class (consisting of classes with more than 100 training samples), the middle class (including classes with 20 to 100 samples), and the tail class (consisting of classes with less than 20 samples).

[0125] Table 5 Top-1 accuracy obtained after integrating different components of TVNet into ResNet-32 on the CIFAR-100-LT dataset with imbalance ratios of 100, 50, and 10.

[0126]

[0127]

[0128] + indicates an increase in the indicator, and - indicates a decrease in the indicator.

[0129] Selection of activation function hyperparameters. In order to study the impact of hyperparameter selection on the performance indicators of the LT-ReLU activation function, the embodiment of the present invention formulated a series of hyperparameter combinations and established a control group. Among them, λ and τ are two key hyperparameters of LT-ReLU. Initially, the embodiment of the present invention sets λ to 0 and τ to 1. Subsequently, while keeping one hyperparameter unchanged, the embodiment of the present invention systematically changes the other hyperparameter to carefully observe the performance change results associated with each hyperparameter value. Detailed data are shown in Tables 6 and 7.

[0130] Table 6 Changes in the top-1 accuracy index (τ=1) when τ is fixed and λ is changed.

[0131]

[0132] Table 7 Changes in top-1 accuracy indicators when λ is fixed and τ is changed (λ=0).

[0133]

[0134] While keeping τ constant, changing λ affects the top-1 accuracy. λ controls the slope of the nonlinear part and needs to be carefully selected to obtain the best model performance in different data sets. The embodiments of the present invention found that λ values ​​between -1 and 1, especially between -0.5 and 1, improve performance, indicating that close to zero or slight positivity will push the function closer to its linear part, thereby increasing the response speed to minority classes. For τ, which affects smoothness, larger values ​​(1 or 2) make the function smoother, stabilizing training by reducing sudden gradient changes. However, smaller values ​​of τ slightly reduce effectiveness, which emphasizes the necessity of balancing smoothness and training stability when choosing τ.

[0135] In order to solve the challenges brought by class imbalance in traditional classification models in the long-tail problem, the embodiment of the present invention proposes a new multi-branch network-Tail-Vigor Network (TVNet). The design purpose of TVNet is to give priority to discovering rare patterns and enhancing features, overcoming the limitations of previous classification networks in making full use of the rich contextual information contained in the image and revealing potential information. The main innovations of the embodiment of the present invention include the introduction of a multi-branch architecture, the simultaneous fusion of ARF and MA-CAM feature fusion modules, and the design of a new ReLU function suitable for the distribution of long-tail data. The implementation of TVNet is simple and intuitive, and a large number of experiments conducted on various benchmark datasets have demonstrated its direct performance improvement and compatibility when combined with other long-tail technologies.

[0136] Figure 4 4 is a diagram of a long-tail image data classification and analysis device according to an embodiment of the present invention. The device includes a first module 410 , a second module 420 , a third module 430 , a fourth module 440 and a fifth module 450 .

[0137] Among them, the first module is used to obtain the target long-tail image, and process the target long-tail image with a long-tail image classification model to obtain the classification result of the target long-tail image; the long-tail image classification model is trained using the following modules: the second module is used to obtain a long-tail data set, and extract features from the long-tail data set using a ResNet backbone network to obtain multi-level hierarchical features; the third module is used to enhance the multi-level hierarchical features using an adaptive context to obtain enhanced multi-level hierarchical features; the fourth module is used to classify the enhanced multi-level hierarchical features based on a linear growth and nonlinear gating mechanism using a ReLU activation function to obtain the classification result of the long-tail data set; the fifth module is used to complete the training of the long-tail image classification model based on the classification result of the long-tail data set.

[0138] Exemplarily, with the cooperation of the first module, the second module, the third module, the fourth module and the fifth module in the device, the embodiment device can implement any of the aforementioned long-tail image data classification methods, that is, obtaining a target long-tail image, processing the target long-tail image with a long-tail image classification model, and obtaining a classification result of the target long-tail image; the long-tail image classification model is trained by the following steps: obtaining a long-tail data set, extracting features from the long-tail data set using a ResNet backbone network, and obtaining multi-level hierarchical features; enhancing the multi-level hierarchical features using adaptive context to obtain enhanced multi-level hierarchical features; classifying the enhanced multi-level hierarchical features based on linear growth and nonlinear gating mechanisms using a ReLU activation function to obtain the classification result of the long-tail data set; and completing the training of the long-tail image classification model based on the classification result of the long-tail data set. The beneficial effects of the present invention are as follows: by adaptively and dynamically adjusting the receptive field size, and dynamically adjusting the size and span of the pooling window to meet the needs of feature distribution of different categories, the feature extraction efficiency of long-tail data is improved; a multi-scale pool and attention mechanism are used to process feature maps of different levels, and the attention mechanism is used to integrate features into feature representation, thereby enhancing the model's understanding and representation capabilities of input data; a new ReLU function is used to cope with the output of dense data and sparse data distribution based on a linear growth mechanism and a nonlinear gating mechanism, respectively, thereby improving sensitivity to data changes; and the accuracy of long-tail image data classification is improved.

[0139] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory;

[0140] The memory stores a program;

[0141] The processor executes the program to perform the aforementioned long-tail image data classification method; the electronic device has the function of carrying and running the long-tail image data classification software system provided by the embodiment of the present invention, for example, a personal computer, a minicomputer, a main frame, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicates with a charged particle tool or other imaging device, etc.

[0142] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the long-tail image data classification method as described above.

[0143] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0144] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned long-tail image data classification method.

[0145] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0146] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0148] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0149] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0151] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0152] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A long-tail image data classification method, characterized in that: include: Acquire a target long-tail image, and process the target long-tail image using a long-tail image classification model to obtain a classification result of the target long-tail image; The long-tail image classification model is trained by the following steps: Obtain a long-tail data set, and use a ResNet backbone network to perform feature extraction on the long-tail data set to obtain multi-level hierarchical features; Adopting adaptive context to enhance the multi-level hierarchical features to obtain enhanced multi-level hierarchical features; The enhanced multi-level hierarchical features are classified using a ReLU activation function based on a linear growth and nonlinear gating mechanism to obtain a classification result of the long-tail data set; According to the classification result of the long-tail data set, the training of the long-tail image classification model is completed.

2. The long-tail image data classification method according to claim 1, characterized in that: The long-tail data set is obtained, and a ResNet backbone network is used to extract features of the long-tail data set to obtain multi-level hierarchical features, including: An input image is extracted from the long-tailed dataset, and according to the dataset type of the long-tailed dataset, a corresponding ResNet backbone network is selected to perform feature extraction using a dynamic pooling layer, wherein the dynamic pooling layer dynamically adjusts the pooling operation parameters by dynamically adjusting the pooling window and according to the feature distribution of the input image.

3. The long-tail image data classification method according to claim 1, characterized in that: The step of enhancing the multi-level hierarchical features by using adaptive context to obtain enhanced multi-level hierarchical features includes: The low-level features, middle-level features and high-level features of the multi-level hierarchical features are obtained, and the features of different levels are enhanced by adaptive context through different numbers of channels to obtain enhanced multi-level hierarchical features. The enhanced multi-level hierarchical features are fused by element-by-element addition. The formula for the enhanced multi-level hierarchical features is: in, To enhance the multi-level features, f i (i=1; 2; 3) is a multi-level hierarchical feature and i represents the level, BN is normalization processing, Conv 3×3 It is 3×3 convolution processing, and ACM is adaptive context processing.

4. The long-tail image data classification method according to claim 3, characterized in that: The process of performing adaptive context enhancement processing on features at different levels through different numbers of channels to obtain enhanced multi-level features also includes: The unenhanced intermediate layer features are fused with the enhanced high-level features using cross-layer connections. The fusion process is: f m =Maxpooling(Conv 1×1 (f2)) Among them, f m is the intermediate layer feature after fusion, Maxpooling is the maximum pooling process, Conv 1×1 It is a 1×1 convolution process and is an intermediate layer feature without enhancement.

5. The long-tail image data classification method according to claim 3, characterized in that: The method of performing enhancement processing on the multi-level hierarchical features by using adaptive context to obtain enhanced multi-level hierarchical features also includes: Multi-scale pooling and attention mechanisms are used to process feature maps at different levels, including average pooling operations on input feature maps of different scales to aggregate contextual information to varying degrees. The process is as follows: x pooled =Avgpool(x,scale) Where Avgpool is the average pooling process, xpooled is the feature map obtained after the average pooling process, and scale = [1, 2, 4].

6. The long-tail image data classification method according to claim 5, characterized in that: The method of enhancing multi-level features by element-by-element addition fusion includes: The enhanced features of the low-level features, the enhanced features of the middle-level features, the enhanced features of the high-level features, and the fused middle-level features are added element by element, and the fused features of different levels are sent to the classifier. The addition process formula is: f′ is the fused multi-level feature, They are the enhanced features of low-level features, the enhanced features of middle-level features, and the enhanced features of high-level features, respectively. m It is the intermediate layer feature after fusion.

7. The long-tail image data classification method according to claim 6, characterized in that: The ReLU activation function is used to classify the enhanced multi-level hierarchical features based on linear growth and nonlinear gating mechanisms to obtain the classification results of the long-tail data set, including: Next, after pooling the feature maps of each layer, convolution operation is performed on the feature maps, and then nonlinear transformation is performed using the ReLU activation function, and the Sigmoid activation function is used to map the output to the range of 0 to 1 to obtain the attention map. The processing process is as follows: Map=Sigmoid(Conv 1×1 (ReLU(Conv 1×1 (X pooled )))) Among them, Map represents the attention map, Sigmoid represents the input mapping process of the Sigmoid activation function, ReLU represents the nonlinear transformation process of the ReLU activation function, and Conv 1×1 represents 1×1 convolution processing, X pooled is the feature map of average pooling; Perform bilinear interpolation on the generated attention map, including adjusting the attention map to the same size as the original input feature map, and concatenating the attention maps generated on all levels along the channel dimension to obtain multi-scale contextual attention features; The ReLU activation function uses the LT-ReLU activation function, which uses a linear growth mechanism and a nonlinear gating mechanism. The linear growth mechanism is used to show linear growth characteristics in dense data areas, and the nonlinear gating mechanism is used to introduce nonlinear elements in sparse data areas. The formula is: f(x)=max(x,0)+(1-max(x,0))·σ(x-λ) Among them, λ is an optimizable adaptive threshold parameter, max(x,0) is a linear growth, and (1-x)·sigmoid(x-λ) is a nonlinear gating. And, use the tanh function to limit the overall output range to [-1, 1], the formula is Where τ is the scaling factor.

8. A long-tail image data classification device, characterized in that: include: The first module is used to obtain a target long-tail image, and process the target long-tail image using a long-tail image classification model to obtain a classification result of the target long-tail image; The long-tail image classification model is trained using the following modules: The second module is used to obtain a long-tail data set, and use a ResNet backbone network to extract features from the long-tail data set to obtain multi-level hierarchical features; The third module is used to enhance the multi-level hierarchical features by using adaptive context to obtain enhanced multi-level hierarchical features; The fourth module is used to classify the enhanced multi-level hierarchical features based on linear growth and nonlinear gating mechanisms using a ReLU activation function to obtain a classification result of the long-tail data set; The fifth module is used to complete the training of the long-tail image classification model according to the classification results of the long-tail data set.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the long-tail image data classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the long-tail image data classification method according to any one of claims 1 to 7.

Images