A long-tail image classification method based on twin network feature diversity enhancement

Abstract

This invention discloses a long-tail image classification method based on Siamese networks with enhanced feature diversity. The method constructs an asymmetric Siamese network: one branch processes the original image, and the other branch processes a mixed sample composed of multiple images, with both branches sharing a backbone network. For the mixed sample branch, a feature diversity enhancement module is designed. Through a fusion process constrained by a lower triangular mask matrix, it guides the rich feature information of the head category to the tail category, thereby enhancing the feature representation of the tail category, while maintaining the feature map spatial structure unchanged throughout the process. The network training employs a hybrid loss function, combining classification loss, contrastive loss, and dynamic gradient balancing loss for collaborative optimization. After training, only the original image branch is used for inference, without adding additional computational burden. This invention effectively alleviates the classification imbalance problem under long-tail data distribution and significantly improves the recognition accuracy of the tail category.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a long-tail image classification method based on Siamese networks with enhanced feature diversity. Background Technology

[0002] Long-tail image classification is a technique aimed at accurately identifying rare tail categories even when training data is extremely imbalanced. In practical applications, data often exhibits a long-tail distribution, meaning a small number of head categories occupy the vast majority of samples, while a large number of tail categories have only a few samples. Existing long-tail classification methods typically employ resampling or reweighting strategies, but these can easily lead to underfitting of the head categories or overfitting of the tail categories. In recent years, contrastive learning methods based on Siamese networks have been introduced into long-tail classification, constructing concatenated mixed samples to balance the input distribution. However, existing Siamese network methods often neglect semantic competition within the concatenated samples during the feature extraction stage, causing the feature extractor to focus on head category regions while suppressing the feature representation of tail categories. Furthermore, traditional feature fusion modules often destroy the spatial structure of images and cannot adapt to the processing of concatenated samples that require preservation of spatial location information.

[0003] The Chinese authorization announcement number is "CN115205594B", and the name is "A classification method for long-tailed image data based on mixed samples". This method first uses the Mixup strategy to perform pixel-level linear interpolation mixing on images with regular sampling and class-balanced sampling. Then, it constructs an expert model containing three different loss functions (such as cross-entropy, LDAM, etc.), extracts features by sharing a backbone network, and fuses the outputs of all experts in the inference stage to make classification decisions. To mitigate the bias caused by the long-tail distribution, this method uses pixel-level mixup. However, this operation leads to semantic aliasing and visual "ghosting" in the generated images, destroying the original spatial structure and local texture details of the images. Especially for the tail category, where samples are scarce, its key features are easily overwhelmed by the information of the head category, resulting in blurred feature extraction. To improve the model's adaptability to different frequency categories, a multi-expert ensemble architecture is used. This leads to redundant model parameters and significantly increased computational overhead during the inference phase, increasing the demand on hardware resources and making it impossible to achieve deployment with zero additional overhead. Furthermore, relying solely on global soft labels generated by the mixing ratio for supervision results in the network lacking the ability to perceive the local semantics of the image, making it difficult to accurately capture the tiny tail category regions in the mixed samples, leading to insufficient feature learning and a lack of discriminative power.

[0004] In summary, designing a novel long-tail image classification method that maintains the integrity of the mixed sample space structure and local semantic independence, fully utilizes the rich semantic information of the head category to enhance the feature representation of the tail category, balances classification accuracy with model complexity in the inference stage, and uses refined supervision signals to solve gradient imbalance in the feature extraction process, thereby achieving accurate classification of long-tailed distributed data, is a problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The technical solution of this invention to solve the above-mentioned technical problems is to provide a long-tail image classification method based on Siamese networks with enhanced feature diversity, comprising:

[0006] Step 1, Data Preprocessing and Category Sorting: Sort the data in descending order based on the number of samples in each category, establish a category priority sequence, and construct a lower triangular mask matrix based on this sequence to constrain the feature flow in subsequent steps;

[0007] Step 2, Construct an asymmetric Siamese network: Construct an asymmetric Siamese network containing a first branch and a second branch, wherein the input of the first branch is the original single image, and the input of the second branch is a mixed sample composed of multiple images;

[0008] Step 3, Shared Feature Extraction: Using the parameter-shared backbone network, feature extraction is performed on the inputs of the first branch and the second branch respectively to obtain the first feature map and the second feature map that retain the original spatial dimensions;

[0009] Step 4, spatially preserved feature diversity enhancement: A feature diversity enhancement module is introduced after the backbone network of the second branch to process the feature map extracted by the second branch; high-dimensional features are mapped to the category semantic space through feature decoupling, and then the learnable fusion matrix constrained by the lower triangular mask matrix is ​​used to perform feature fusion between channels. Finally, the enhanced feature map is output through residual connection, and the spatial size of the feature map remains unchanged throughout the process.

[0010] Step 5, Construct a hybrid loss function: Construct a total loss function for joint optimization of the network. The total loss function includes at least: a classification loss applied to the first branch, a contrast loss based on the enhanced feature map, and a dynamic gradient balancing loss that directly supervises the intermediate output of the feature diversity enhancement module.

[0011] Step 6, train the network model: train the asymmetric Siamese network using the total loss function. After training, use only the first branch to classify and predict the test image.

[0012] Furthermore, step 1 specifically includes:

[0013] The data collection sets the number of samples in each category, sorts them in descending order of quantity, and reassigns the category indexes so that the index values ​​reflect the sample richness.

[0014] Construct a mask matrix of size C×C, where C is the total number of categories; in this mask matrix, the element value is 1 when the row index i is greater than or equal to the column index j, and 0 otherwise, thus forming a lower triangular matrix structure.

[0015] Furthermore, in step 2, an asymmetric twin network is constructed:

[0016] First branch: The input to this branch is a raw single image Iorg randomly sampled from the dataset, which is mainly used to learn the global discriminative features of the image;

[0017] The second branch: The input to this branch is an artificially constructed mixed sample Istitch (randomly select 4 images of different categories, uniformly scale them to half the size of the original image, and stitch them together according to a 2×2 spatial grid).

[0018] Construct a multi-hotspot label vector: Based on the categories contained in the mixed sample, construct a multi-hotspot vector t∈{0,1}C of length C; if the mixed sample contains the c-th class target, the value of vector t at the c-th position is 1, otherwise it is 0.

[0019] Furthermore, in step 3, the specific process of extracting features using the shared backbone network is as follows: the output of the first branch... The output of the second branch They are fed into a deep convolutional neural network with shared weights;

[0020] For the first branch, the output feature map is denoted as... , dimension ;

[0021] For the second branch, the output feature map is denoted as... , dimension ;

[0022] in, For batch size, The number of feature channels, and These represent the height and width of the feature map, respectively.

[0023] Furthermore, the feature diversity enhancement module in step 4 performs the following operations in sequence:

[0024] Feature decoupling: The second feature map with D channels is mapped to a category-aware feature map with the number of channels equal to the total number of categories C through the first 1×1 convolutional layer. ;

[0025] Mask fusion: Define a learnable fusion matrix: ; and the mask matrix in the feature decoupling step. Performing the Hadamard product yields the constrained fusion matrix: ;use right Perform matrix multiplication along the channel dimension to obtain the fused features: ;

[0026] Feature recovery and residual connection: The number of channels in the fused feature map is restored from C to D through a second 1×1 convolutional layer, and the result is added to the input second feature map to obtain the enhanced feature map. .

[0027] Furthermore, in step 5, constructing the hybrid objective function includes:

[0028] In the first branch classification task, for Perform global average pooling to reduce the spatial dimension Compress to To obtain the global feature vector Feed it into the classifier and calculate the Logit-adjusted cross-entropy loss. This is used to optimize the model's basic classification performance;

[0029] In the second branch, a contrastive learning task is performed on the enhanced features. Perform adaptive average pooling to force the space size to be adjusted. (Corresponding to the grid layout during stitching), the feature map is obtained:

[0030] ,

[0031] Then, spatial segmentation is performed, dividing it into 4 independent feature vectors:

[0032] ,

[0033] These correspond to the four sub-images before stitching; the vectors are mapped to the projection space, and the loss is compared with the supervised calculation of features from the T1 branch and the in-memory database. :

[0034] ,

[0035] in The projected feature vectors For the positive sample set, Temperature coefficient;

[0036] Finally, a feature diversity enhancement module is used to assist the supervision task, extracting the intermediate class-aware features from step 4 and performing global average pooling to obtain the prediction vector. Using the multi-hotspot labels generated in step 2, calculate the dynamic gradient balancing loss. :

[0037] ,

[0038] in For dynamic weights based on the reciprocal of class frequency, Here, C is the Sigmoid activation function, and C represents the number of classes.

[0039] The technical solution of the present invention has the following beneficial effects:

[0040] This invention designs a Siamese network architecture based on feature diversity enhancement. By introducing a feature diversity enhancement module into the comparison branch, the spliced ​​samples are processed from two dimensions: spatial preservation and semantic fusion. This can effectively improve the ability of existing long-tail classification networks in terms of tail category feature capture, semantic decoupling of spliced ​​samples, and feature expression diversity, and solve the bottleneck of traditional Siamese networks suppressing tail categories when processing mixed samples.

[0041] This invention designs a mask fusion unit that constrains the feature fusion process by using a lower triangular mask matrix arranged in descending order of class frequency. This helps the network block the propagation of tail noise to the head during feature interaction, while strengthening the absorption of general features of the head by the tail category and improving the feature robustness of the tail category.

[0042] This invention designs a hybrid objective function that includes dynamic gradient balancing loss. By introducing dynamic gradient balancing loss to directly supervise the intermediate layer of the feature enhancement module, the network can be explicitly forced to focus on all class regions in the spliced ​​samples through a multi-label supervision mechanism, reducing semantic loss during feature extraction and thus maintaining good classification performance even when facing extreme long-tailed distributions. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0044] Figure 1 A flowchart of the long-tail image classification method based on Siamese networks with enhanced feature diversity provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the overall framework of the asymmetric twin network provided by the present invention;

[0046] Figure 3 A schematic diagram of the feature diversity enhancement module structure provided by the present invention;

[0047] Figure 4 This is a schematic diagram of the asymmetric feature flow control based on the mask matrix described in this invention;

[0048] Figure 5 This is a schematic diagram of the input of image data for a twin network provided by the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the spatial semantic correspondence of the spliced ​​samples during the feature extraction process described in this invention.

[0050] Figure 7 This is a schematic diagram of the T2 branch network structure provided by the present invention;

[0051] Figure 8 This is a schematic diagram of feature channel changes in a specific example 2 provided by the present invention. Detailed Implementation

[0052] This invention proposes a long-tail image classification method based on Siamese networks with enhanced feature diversity, aiming to achieve accurate classification of long-tailed distributed data and thereby improve the recognition accuracy of rare tail categories.

[0053] The long-tail image classification method based on Siamese networks with enhanced feature diversity proposed in this invention will be described below in specific embodiments:

[0054] Example 1:

[0055] In the technical solution of this embodiment, such as Figure 1 As shown, a long-tail image classification method based on Siamese networks with enhanced feature diversity includes:

[0056] Step 1, Data Preprocessing and Category Sorting: Sort the data in descending order based on the number of samples in each category, establish a category priority sequence, and construct a lower triangular mask matrix based on this sequence to constrain the feature flow in subsequent steps;

[0057] Step 2, Construct an asymmetric Siamese network: Construct an asymmetric Siamese network containing a first branch and a second branch, wherein the input of the first branch is the original single image, and the input of the second branch is a mixed sample composed of multiple images;

[0058] Step 3, Shared Feature Extraction: Using the parameter-shared backbone network, feature extraction is performed on the inputs of the first branch and the second branch respectively to obtain the first feature map and the second feature map that retain the original spatial dimensions;

[0059] Step 4, spatially preserved feature diversity enhancement: A feature diversity enhancement module is introduced after the backbone network of the second branch to process the feature map extracted by the second branch; high-dimensional features are mapped to the category semantic space through feature decoupling, and then the learnable fusion matrix constrained by the lower triangular mask matrix is ​​used to perform feature fusion between channels. Finally, the enhanced feature map is output through residual connection, and the spatial size of the feature map remains unchanged throughout the process.

[0060] Step 5, Construct a hybrid loss function: Construct a total loss function for joint optimization of the network. The total loss function includes at least: a classification loss applied to the first branch, a contrast loss based on the enhanced feature map, and a dynamic gradient balancing loss that directly supervises the intermediate output of the feature diversity enhancement module.

[0061] Step 6, train the network model: train the asymmetric Siamese network using the total loss function. After training, use only the first branch to classify and predict the test image.

[0062] Furthermore, step 1 specifically includes:

[0063] The data collection sets the number of samples in each category, sorts them in descending order of quantity, and reassigns the category indexes so that the index values ​​reflect the sample richness.

[0064] Construct a mask matrix of size C×C, where C is the total number of categories. In this mask matrix, the element value is 1 when the row index i is greater than or equal to the column index j, and 0 otherwise, thus forming a lower triangular matrix structure. This lower triangular mask matrix will be used in subsequent steps to ensure that feature information flows only from high-frequency categories to low-frequency categories, preventing noise from tail categories from interfering with the feature learning of head categories.

[0065] Specifically, let's assume a long-tailed image dataset. Include There are several categories. First, the sample frequency of each category in the training set is counted. And perform a descending sort operation to make the rearranged category index satisfy ,

[0066] In other words, the smaller the index, the richer the category samples. Based on this ordered index, a dimension is initialized. lower triangular mask matrix Its element definition is as follows:

[0067] ,

[0068] In subsequent feature fusion steps, this matrix ensures that features from high-frequency categories are allowed to flow to low-frequency categories, thereby physically blocking the reverse propagation of tail noise features to head categories and thus guaranteeing the stability of feature fusion.

[0069] Furthermore, in step 2, an asymmetric twin network is constructed, such as... Figure 2 As shown:

[0070] First branch: The input to this branch is a raw single image Iorg randomly sampled from the dataset, which is mainly used to learn the global discriminative features of the image;

[0071] The second branch: The input to this branch is an artificially constructed mixed sample Istitch (randomly select 4 images of different categories, uniformly scale them to half the size of the original image, and stitch them together according to a 2×2 spatial grid).

[0072] Construct a multi-hotspot label vector: Based on the categories contained in the mixed sample, construct a multi-hotspot vector t∈{0,1}C of length C; if the mixed sample contains the c-th class target, the value of vector t at the c-th position is 1, otherwise it is 0.

[0073] The specific process involves randomly selecting four images from different categories. The corresponding tag set is After scaling these four images uniformly, according to The grid layout is stitched together to form a complete image. To achieve semantic supervision of the stitched samples, the set of categories contained in the stitched image is used: Construct a length of Multi-hot vector The definition is as follows:

[0074] .

[0075] Furthermore, in step 3, the specific process of extracting features using the shared backbone network is as follows: the input of the first branch... Input of the second branch They are fed into a deep convolutional neural network with shared weights;

[0076] For the first branch, the output feature map is denoted as... , dimension ;

[0077] For the second branch, the output feature map is denoted as... , dimension ;

[0078] in, For batch size, The number of feature channels, and These represent the height and width of the feature map, respectively.

[0079] Specifically, the key to this step is removing the global pooling layer at the end of the backbone network while preserving the complete spatial dimension to maintain the local semantic independence of the 2×2 layout in xstitch. If global pooling were performed at this point, it would cause irreversible aliasing of features from different categories, disrupting subsequent feature decoupling.

[0080] Furthermore, such as Figure 3 As shown, the feature diversity enhancement module in step 4 performs the following operations in sequence:

[0081] Feature decoupling: The second feature map with D channels is mapped to a category-aware feature map with the number of channels equal to the total number of categories C through the first 1×1 convolutional layer. ;

[0082] At this time, tensor Defined as a category-aware feature, its first The response graph of the first channel corresponds to the first... The activation intensity of each category. This intermediate variable. It will serve as the direct supervision object for any subsequent additions to the loss function.

[0083] Mask fusion: Define a learnable fusion matrix: ; and the mask matrix in the feature decoupling step. Performing the Hadamard product yields the constrained fusion matrix: ;use right Perform matrix multiplication along the channel dimension to obtain the fused features: The operation, while maintaining the spatial structure (H,W) unchanged, enables the features of the tail category region in the spliced ​​sample to absorb the semantic information of the head category. This is equivalent to using the rich semantic patterns of the head category (such as common textures and shapes) to fill the sparse feature space of the tail category, thereby enhancing the robustness of the features.

[0084] Feature recovery and residual connection: The number of channels in the fused feature map is restored from C to D through a second 1×1 convolutional layer. A residual connection is then introduced to add the fused feature map to the second input feature map, resulting in the enhanced feature map. Residual connection: Output features Will replace the original Proceeding to the subsequent comparative learning module. Its spatial dimension remains unchanged. .

[0085] Specifically, the feature decoupling unit first maps high-dimensional features to the category semantic space; then, the mask fusion unit uses a constrained fusion matrix to control the feature flow direction, forcing the head category features to flow to the tail category features; finally, the residual connection unit fuses the enhanced features and the original features, outputting diversity-enhanced features while maintaining the spatial resolution.

[0086] Furthermore, in step 5, constructing the hybrid objective function includes:

[0087] In the first branch classification task, for Perform global average pooling to reduce the spatial dimension Compress to To obtain the global feature vector Feed it into the classifier and calculate the Logit-adjusted cross-entropy loss. This is used to optimize the model's basic classification performance;

[0088] The parameters in the formula are defined as follows: For the input sample, Its corresponding real category label, Total number of categories; This represents the corresponding category output by the model's classification layer. The original Logit value; Represents the categories in the training dataset The total number of samples (i.e., prior frequencies); For prior compensation terms, defined as This introduces category frequency information, making the tail rare categories ( The smaller the probability, the larger the Logit compensation, thus offsetting the prediction bias caused by the long-tailed distribution; The model represents the categories The attention coefficient. In this embodiment, this coefficient is used to adjust the gradient weights of different categories in backpropagation, and is typically set to... .

[0089] In the second branch, a contrastive learning task is performed on the enhanced features. Perform adaptive average pooling to force the space size to be adjusted. (Corresponding to the grid layout during stitching), the feature map is obtained: Then, spatial segmentation is performed, dividing it into 4 independent feature vectors: These correspond to the four sub-images before stitching; the vectors are mapped to the projection space, and the loss is compared with the supervised calculation of features in the T1 branch and the memory database. :

[0090] ,

[0091] in The projected feature vectors For the positive sample set, Temperature coefficient;

[0092] Finally, a feature diversity enhancement module is used to assist the supervision task, extracting the intermediate class-aware features from step 4 and performing global average pooling to obtain the prediction vector. Using the multi-hotspot labels generated in step 2, calculate the dynamic gradient balancing loss. :

[0093] ,

[0094] in Here, C is the sigmoid activation function, and C represents the number of classes. The dynamic balancing weights are defined as follows:

[0095] (Based on the inverse effective sample count), this loss forces the feature diversity enhancement module to... The convolutional layer must accurately decouple all categories contained in the stitched image. This is due to the introduction of... The model applies a larger gradient penalty to tail categories during backpropagation to prevent tail features from being "submerged" during the feature extraction stage.

[0096] Furthermore, in step 6, the network model is trained: the final model training optimization objective is defined as:

[0097] ,

[0098] in and These are hyperparameters used to balance the weights of the various loss terms. During backpropagation, The gradient will directly update the feature decoupling unit and the backbone network, forcing the backbone network to retain all occurrences of category semantics when extracting features from the stitched image.

[0099] During the inference phase, the model uses only the first branch for image classification prediction. At this time, the second branch and its contained feature diversity enhancement module do not participate in the computation, achieving an efficient deployment of "enhancement during training and zero additional overhead during inference".

[0100] Example 2:

[0101] A long-tail image classification method based on Siamese networks with enhanced feature diversity, such as Figure 8As shown, the specific steps include the following:

[0102] This embodiment uses the CIFAR-100-LT dataset with an imbalance factor (IF) of 100 as an example, and elaborates on the specific implementation process of the method described in this invention in conjunction with an improved asymmetric Siamese network architecture. The network input image size is set to... Pixel.

[0103] Step 1, as follows Figure 5 As shown, data preprocessing and category sorting: This step aims to construct long-tailed distribution data and establish prior constraints based on sample frequency.

[0104] (1) Construction of long-tail data: The CIFAR-100 dataset contains 100 categories, with 500 training images for each category initially. Based on the exponential decay model of the long-tail distribution, the first... Number of samples in each category Set as:

[0105] ,

[0106] Among them, the total number of categories Maximum number of samples Imbalance factor .

[0107] (2) Category Reordering: Count the number of samples in each category after generation and perform a descending sort operation. Reallocate the category index. This makes the index The index corresponds to the head category with the most samples. The tail category with the fewest corresponding samples.

[0108] (3) Mask matrix initialization: Based on the sorted indices, construct a mask matrix of size... lower triangular mask matrix Define matrix elements. If and only if (i.e. row index (column index), otherwise This matrix will physically block the propagation of noise information from the low-frequency category to the high-frequency category during subsequent manifold alignment in the feature space.

[0109] Step 2, Construct an asymmetric Siamese network: Construct an asymmetric input queue containing a first branch (T1) and a second branch (T2) to balance original distribution learning and balanced feature mining;

[0110] (1) T1 branch (original stream): The input is a batch of images randomly sampled from the long-tail training set. and its corresponding single tag This branch is used to maintain the model's discriminative boundary for the original long-tailed distribution.

[0111] (2) T2 branch (balanced flow): The input is an artificially constructed hybrid spliced ​​sample. .

[0112] First, sampling and stitching are performed. For each sample in the batch, four images of different categories are randomly selected. Scaled uniformly to using bilinear interpolation Pixels. Then, in order of top left, top right, bottom left, bottom right... The grid order is used to stitch these four sub-images together into a complete image. image.

[0113] Furthermore, define the label vector. Let the category indices of these four subgraphs be respectively... Then the vector The element at these four index positions has a value of 1, and the rest are 0. This vector accurately describes the semantic composition of the mixed samples and is used for subsequent dynamic gradient balancing supervision.

[0114] Step 3: Feature Extraction Using a Shared Backbone Network: This step utilizes a parameter-shared deep convolutional neural network to extract spatial features from the image. ResNet-32 is selected as the backbone network. To accommodate the local semantic independence of the stitched samples, the global average pooling layer and fully connected layer at the end of the network are removed, retaining only the convolutional layer structure.

[0115] First, feature extraction is performed, and Simultaneously fed into the backbone network. Since the total downsampling stride of ResNet-32 on the CIFAR dataset is typically 4, the spatial size of the output feature map is... (Right now ).

[0116] The following are the output dimensions of the two branches after feature extraction.

[0117] T1 branch output characteristics: ;

[0118] T2 branch output characteristics: ;

[0119] Where 64 represents the number of output channels of the last convolutional layer in ResNet-32. At this point, Feature maps preserve space The semantic layout, i.e., the top left corner. The regional features strictly correspond to the semantic information of the first subgraph, and the spatial topological relationships are fully preserved.

[0120] Step 4, Introducing a Feature Diversity Enhancement Module: This step introduces a feature diversity enhancement module in the T2 branch. While maintaining the spatial structure, it leverages the rich semantics of the head category to enhance the feature representation of the tail category. The spatial semantic correspondence of the concatenated samples during feature extraction is as follows: Figure 6 As shown, the T2 branch network structure is as follows: Figure 7 As shown, the feature diversity enhancement module enhances high-dimensional features in the T2 branch. Processing is performed. For the 64-dimensional channels of ResNet-32 and the 100 categories of CIFAR-100, the module computation is as follows:

[0121] (1) Feature decoupling unit: using the first Convolutional layer (weights) Input features Mapping from the channel dimension to the category semantic dimension:

[0122] ,

[0123] At this time, tensor The The channel represents the first The activation heatmap of each category in space. This intermediate variable... It was introduced to calculate the dynamic gradient balance loss.

[0124] (2) Mask fusion unit: Define a learnable fusion matrix Applying the mask matrix from step 1 to perform Hadamard product constraints, the asymmetric feature flow control of the mask matrix is ​​as follows: Figure 4 As shown, the restricted matrix is ​​obtained:

[0125] ,

[0126] use right Perform matrix multiplication along the channel dimension:

[0127] ,

[0128] This step achieves asymmetric feature interaction; for the tail category, its features incorporate the general patterns of the head category, while the head category is unaffected by tail noise. Since the number of parameters is only [amount missing], [further details about parameter count and parameter count are needed]. With only one parameter, this module is extremely lightweight and adds almost no computational burden.

[0129] (3) Residual connection unit: using the second Convolutional layer (weights) The features are restored to their original 64-dimensional channels, and residual connections are introduced:

[0130] ,

[0131] Output For semantic enhancement and complete preservation Characteristic diagram of spatial structure.

[0132] Step 5, Construct a hybrid objective function: Design differentiated post-processing paths and loss functions for different levels of network output features to achieve multi-task collaborative optimization.

[0133] (1) T1 branch classification path: global pooling and Logit adjustment:

[0134] right ( Perform global average pooling to compress the spatial dimension to The eigenvectors are obtained as follows:

[0135] ,

[0136] Then Input a linear classifier to obtain Logit To eliminate the classification boundary offset caused by the long-tail distribution, LogitAdjustment(LA)Loss is used:

[0137] ,

[0138] in For temperature coefficient, The sample frequency represents the class. This loss forces the model to output a larger Logit value for the tail classes to offset prior probability bias.

[0139] (2) T2 branch comparison path: Adaptive pooling and spatial partitioning:

[0140] Adaptive pooling: In order to... Feature maps and input Alignment of puzzle layout enhances features Perform adaptive average pooling, forcing the output size to be... :

[0141] ,

[0142] This operation is mathematically equivalent to... The feature map is divided into four The sub-regions are defined, and the mean is calculated within each sub-region.

[0143] Then, the feature map is spatially split. It is divided into 4 independent feature vectors in the spatial dimension.

[0144] ,

[0145] Each vector has a dimension of 64, corresponding to the four sub-images of the original stitched image.

[0146] These vectors are then mapped into the embedding space using a projection head (MLP). Supervised contrastive loss is then applied. Conduct comparative learning.

[0147] ,

[0148] The features of these subgraphs are brought closer to similar features in the T1 branch and the MemoryBank, while distancing them from dissimilar features.

[0149] (3) Feature enhancement auxiliary path: Dynamic gradient balancing supervision:

[0150] First, extract the intermediate features from step 4. ( Perform global average pooling to obtain the class prediction vector:

[0151] ,

[0152] Then, utilize the multi-hotspot tags generated in step 2. Calculate the dynamic gradient balancing loss (DGBL):

[0153] ,

[0154] in This is the Sigmoid function.

[0155] ,

[0156] The weights are based on the inverse effective sample number. This loss function forces the decoupling layer of the DEM module to accurately identify all categories contained in the stitched image, preventing tail features from being overwhelmed.

[0157] Step 6, Train the network model: Construct the final joint optimization objective:

[0158] ,

[0159] In this embodiment, a balance coefficient is set:

[0160] ,

[0161] The network was trained end-to-end using the SGD optimizer. Momentum was set to 0.9, and weight decay was... The initial learning rate was set to 0.1, and a cosine annealing strategy was used to decay it to 0 over 200 epochs.

[0162] During the reverse propagation process, The resulting gradient flow will directly update the parameters of the feature diversity enhancement module and the backbone network, correcting the sensitivity of the feature extractor to the tail category.

[0163] After training, only the T1 branch (ResNet-32+ classifier) ​​is retained for inference. At this point, the T2 branch and the feature diversity enhancement module are removed, the model parameter count is the same as the standard ResNet-32, and there is no loss in inference speed.

[0164] Table 1. Top-1 accuracy of ResNet-32 under different imbalance factors in CIFAR 100-LT

[0165]

[0166] As shown in Table 1, the proposed method demonstrates superior performance on the CIFAR100-LT dataset under the same Top-1 accuracy metric. Particularly in severe scenarios of extreme imbalance (IF=100) and moderate imbalance (IF=50), our model achieves accuracies of 56.2% and 61.3%, respectively, surpassing state-of-the-art methods such as GLMC and ResLT, achieving the best results. This fully demonstrates the effectiveness of the proposed feature diversity enhancement module and dynamic gradient balancing strategy in addressing severe long-tailed distribution problems. Although it achieves suboptimal performance in mild imbalance (IF=10), it still maintains a highly competitive accuracy (69.5%). Overall, this method significantly improves the model's ability to handle scarce samples while maintaining overall performance.

[0167] In the above embodiments, the implementations of convolution, ResNet-32 backbone network, activation functions, global pooling, adaptive pooling, residual connections, matrix multiplication operations, and element-wise multiplication are algorithms well known to those skilled in the art, and the specific processes and methods can be found in relevant textbooks or technical documents.

[0168] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A long-tail image classification method based on Siamese networks with enhanced feature diversity, characterized in that, include: Step 1, Data Preprocessing and Category Sorting: Sort the data in descending order based on the number of samples in each category, establish a category priority sequence, and construct a lower triangular mask matrix based on this sequence to constrain the feature flow in subsequent steps; Step 2, Construct an asymmetric Siamese network: Construct an asymmetric Siamese network containing a first branch and a second branch, wherein the input of the first branch is the original single image, and the input of the second branch is a mixed sample composed of multiple images; Step 3, Shared Feature Extraction: Using the parameter-shared backbone network, feature extraction is performed on the inputs of the first branch and the second branch respectively to obtain the first feature map and the second feature map that retain the original spatial dimensions; Step 4, spatially preserved feature diversity enhancement: A feature diversity enhancement module is introduced after the backbone network of the second branch to process the feature maps extracted by the second branch; High-dimensional features are mapped to the category semantic space through feature decoupling, and then the learnable fusion matrix constrained by the lower triangular mask matrix is ​​used to perform feature fusion between channels. Finally, the enhanced feature map is output through residual connection, and the spatial size of the feature map remains unchanged throughout the process. Step 5, Construct a hybrid loss function: Construct a total loss function for jointly optimizing the network. The total loss function includes at least: a classification loss applied to the first branch, a contrast loss based on the enhanced feature map, and a dynamic gradient balancing loss that directly supervises the intermediate output of the feature diversity enhancement module. Step 6, train the network model: train the asymmetric Siamese network using the total loss function. After training, use only the first branch to classify and predict the test image. Step 1 specifically includes: The data collection sets the number of samples in each category, sorts them in descending order of quantity, and reassigns the category indexes so that the index values ​​reflect the sample richness. Construct a mask matrix of size C×C, where C is the total number of categories; in this mask matrix, the element value is 1 when the row index i is greater than or equal to the column index j, and 0 otherwise, thus forming a lower triangular matrix structure; The feature diversity enhancement module in step 4 performs the following operations in sequence: Feature decoupling: The second feature map with D channels is mapped to a category-aware feature map with the number of channels equal to the total number of categories C through the first 1×1 convolutional layer. ; Mask fusion: Define a learnable fusion matrix: ; and combine it with the mask matrix Performing the Hadamard product yields the constrained fusion matrix: ;use right Perform matrix multiplication along the channel dimension to obtain the fused features: ; Feature recovery and residual connection: The number of channels in the fused feature map is restored from C to D through a second 1×1 convolutional layer, and the result is added to the input second feature map to obtain the enhanced feature map. ; For batch size, The number of feature channels, and These represent the height and width of the feature map, respectively.

2. The method according to claim 1, characterized in that, In step 2, an asymmetric twin network is constructed: First branch: The input to this branch is a raw single image Iorg randomly sampled from the dataset, used to learn the global discriminative features of the image; The second branch: The input to this branch is the artificially constructed mixed sample Istitch; Construct a multi-hotspot label vector: Based on the categories contained in the mixed sample, construct a multi-hotspot vector t∈{0,1}C of length C; if the mixed sample contains the c-th class target, the value of vector t at the c-th position is 1, otherwise it is 0.

3. The method according to claim 1, characterized in that, In step 3, the specific process of extracting features using the shared backbone network is as follows: the output of the first branch... The output of the second branch They are fed into a deep convolutional neural network with shared weights; For the first branch, the output feature map is denoted as... , dimension ; For the second branch, the output feature map is denoted as... , dimension .

4. The method according to claim 3, characterized in that, In step 5, constructing the hybrid objective function includes: In the first branch classification task, for Perform global average pooling to reduce the spatial dimension Compress to To obtain the global feature vector Feed it into the classifier and calculate the Logit-adjusted cross-entropy loss. This is used to optimize the model's basic classification performance; In the second branch, a contrastive learning task is performed on the enhanced features. Perform adaptive average pooling to force the space size to be adjusted. The feature map is obtained as follows: ， Then, spatial segmentation is performed, dividing it into 4 independent feature vectors: ， These correspond to the four sub-images before stitching; the vectors are mapped to the projection space, and the loss is compared with the supervised calculation of features from the T1 branch and the in-memory database. : ， in The projected feature vectors For the positive sample set, Temperature coefficient; Finally, a feature diversity enhancement module is used to assist the supervision task, extracting the intermediate class-aware features from step 4 and performing global average pooling to obtain the prediction vector. Using the multi-hotspot labels generated in step 2, calculate the dynamic gradient balancing loss. : ， in For dynamic weights based on the reciprocal of class frequency, Here, C is the Sigmoid activation function, and C represents the number of classes.

Images