CA-SwinT-based hierarchical image classification method and system
The CA-SwinT-based method addresses the challenge of complex hierarchical image classification by employing SwinTransformer and dual cross-attention to enhance feature extraction and label mapping, resulting in improved accuracy and efficiency.
Patent Information
- Application Number
- GB2025018047
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-11-06
- Filing Date
- 2025-10-23
- Publication Date
- 2026-11-11
AI Technical Summary
Existing hierarchical image classification methods struggle to accurately capture complex hierarchical relationships between image labels due to limitations in modeling label dependencies and correlations, particularly in multi-label scenarios, leading to suboptimal recognition performance.
A CA-SwinT-based method utilizing SwinTransformer and dual cross-attention mechanisms to extract fine-grained features and construct comprehensive feature mappings between images and labels, reducing model complexity and enhancing accuracy through local window attention and bidirectional information exchange.
The method achieves improved feature representation, accuracy, and efficiency in hierarchical classification by effectively capturing global and detailed image information while reducing computational complexity and redundancy.
Smart Images

Figure 00000001_0000 
Figure 00000002_0000
Abstract
Description
Technical field The present invention belongs to the field of artificial intelligence technology, and specifically relates to a CA-SwinT-based hierarchical image classification method and system. Background art Hierarchical image classification tasks target multi-label images with hierarchical structures, which play a crucial role in many practical application scenarios. To facilitate consumers in easily finding products, e-commerce platforms have established finely divided product classification systems. Within these systems, broad categories (e.g., electronic products) branch out into more detailed sub-categories (such as laptop computers). In the hierarchical classification of images for design patents, an image labeled with the sub-class of bed can also be categorized under the broader class, furniture. In the biological classification of animals in nature, a Leonberger can be specifically classified under the category of dog and more broadly under the parent class of animal. All these examples demonstrate the hierarchical relationships between labels. In recent years, with significant breakthroughs in artificial intelligence technology, the fields of image classification and segmentation have received unprecedented attention. However, due to the more complex hierarchical dependency structures presented by image labels in the real world, hierarchical image classification tasks face even greater challenges. This lies not only in the task difficulty far exceeding that of traditional single-label classification but also in the need to deeply understand and precisely process the hierarchical relationships among multiple labels. Regarding the hierarchical image classification problem, previous research often employed traditional convolutional neural network methods for recognition. Although these methods achieved relatively good results, they primarily focused on analyzing the correlation between images and labels, failing to fully consider the structured relationships between labels and requiring multiple classifiers to accomplish multi-label hierarchical classification. To address this shortcoming, subsequent research introduced methods based on CNN-RNN. While CNN-RNN methods improved in modeling label correlations, they relied on global image features and global correlations between labels, limiting feature descriptive power and the deep understanding of potential label dependencies. With the successful application of attention mechanisms in various fields, there has been increasing focus on utilizing these technologies to solve hierarchical image classification problems. Furthermore, hierarchical image classification methods based on attention mechanisms have also achieved better recognition performance. However, despite these favorable advancements, shortcomings still exist in fully capturing the associative links and accuracy between images and labels. Summary of the invention The present invention aims to utilize a lightweight framework to effectively improve the accuracy of hierarchical classification, and better solve the shortcomings of the above-mentioned technologies. To this end, an object of the present invention is to provide a CA-SwinT-based hierarchical image classification method, wherein the method introduces main mechanisms of SwinTransformer and a dual cross-attention mechanism. A SwinT module, based on the SwinTransformer, effectively reduces model parameters, lowers model complexity, and extracts rich, fine-grained feature information from images. Simultaneously, the dual cross-attention mechanism can better construct a more comprehensive feature mapping relationship between images and labels, which thereby efficiently identifies multi-level labels of images. To address the above technical problem, the present invention is implemented as follows: Embodiments of the present invention provide a CA-SwinT-based hierarchical image classification method, wherein the method comprises: 2 SI, Acquiring an original image and performing preprocessing on the original image; S2, Extracting features from the image via a SwinT module; S3, Fully integrating the extracted features and classification label information by a dual cross-attention mechanism, and constructing a fine-grained mapping between the image and hierarchical labels; S4, Generating corresponding labels through a label generation module based on the fine-grained mapping from the step S3, and updating the classification label information; Additionally, the CA-SwinT-based hierarchical image classification method according to the present invention may further have the following additional technical features: In some of the embodiments, the step of extracting features from the image via a SwinT module in the step S2 comprises: S201, Patch partitioning processing, comprises dividing the image into a series of small patches; S202, Linear embedding processing, comprises flattening each of the patches and processing it through a fully connected linear embedding layer to transform a pixel value of each of the patches into a high-dimensional feature vector; S203, Two-level Swin module processing, comprises sequentially inputting processing results from the S202 into the two-level Swin module for feature extraction; S204, Patch merging processing, comprises performing down-sampling on the image to half height and width thereof and to double channels thereof; S205, Two-level Swin processing, comprises sequentially inputting processing results from the S204 into the two-level Swin module for feature extraction; S206, Linear embedding processing, comprises applying a linear transformation to processing results from the S205. In some of the embodiments, the processing procedure of each of the stages of the Swin module in the step S203 and step S205 comprises: Layer normalization processing, comprises standardizing data; Window-based multi-head self-attention processing, comprises dividing all pixels of the input image into multiple windows, calculating correlation between each of the 3 pixels and the other pixels within each of the windows to capture local features; Residual connection processing, followed by layer normalization processing on the results, wherein the residual connection processing comprises directly adding the input features to self-attention processing results from the previous step, forming a residual connection; Multilayer perceptron processing, comprises performing complex transformation and combination of the features through fully connected layers and nonlinear activation functions to enhance feature representation capability of a model; Shifted window-based multi-head self-attention processing, comprises dividing all the pixels of the input image into the multiple windows, while shifting the divided windows rightwards and downwards by the several pixels during division, and calculating the correlation between each of the pixels and the other pixels within each of the windows to achieve information interaction between the different windows; Residual connection processing, followed by layer normalization processing on the results, wherein the residual connection processing comprises directly adding the input features to the self-attention processing results from the previous step, forming the residual connection; Multilayer perceptron processing, comprises performing complex transformation and combination of the features through the fully connected layers and the nonlinear activation functions to enhance the feature representation capability of the model. In some of the embodiments, the step of fully integrating the extracted features and classification label information by a dual cross-attention mechanism, and constructing a fine-grained mapping between the image and hierarchical labels in S3 comprises: S301, Taking the features of each of the patches extracted by the SwinT module as keys and values, and taking the current classification label information as queries, performing multi-head attention mechanism processing, merging the processing results, and then performing layer normalization processing to obtain output results of the first cross-attention mechanism; S302, Taking the features of each of the patches extracted by the SwinT module as the keys and the values, and taking the current classification label information as queries, performing multi-head attention mechanism processing, merging the processing results, and then performing layer normalization processing to obtain output results of the second cross-attention mechanism. In some of the embodiments, the step of generating the corresponding labels through the label generation module comprises: S401, Multilayer perceptron processing, comprises processing the input data through the two fully connected layers to enhance the feature representation capability; S402, Classification processing, comprises calculating confidence via a softmax function to achieve classification. Embodiments of the present invention further provide a CA-SwinT-based hierarchical image classification system, comprises the system comprises: A preprocessing module, configured to receive the original image and perform preprocessing on the original image; A feature extraction module, configured to perform feature extraction on the image via the SwinT module; A dual cross-attention module, configured to fully integrate the extracted features and the classification label information by utilizing the dual cross-attention mechanism, and to construct the fine-grained mapping between the image and the hierarchical labels; A label generation module, configured to generate the corresponding labels based on the fine-grained mapping obtained by the dual cross-attention module, and to update the classification label information. Additionally, the CA-SwinT-based hierarchical image classification system according to the present invention may further have the following additional technical features: In some of the embodiments, the feature extraction module comprises, connected sequentially: A patch partitioning layer, configured to divide the image into a series of small patches; A linear embedding layer, configured to flatten each of the patches and process it through a fully connected linear embedding layer to transform a pixel value of each of the patches into a high-dimensional feature vector; A two-level Swin module, configured to input processing results from the preceding linear embedding layer for feature extraction; A patch merging module, configured to halve height and width of the image and double channels thereof through down-sampling; An another two-level Swin module, configured to perform feature extraction on processing results from the preceding patch merging module; A linear embedding module, configured to apply a linear transformation to processing results from the preceding two-level Swin module. In some of the embodiments, each level of the Swin module comprises, connected sequentially: A layer normalization layer, configured to standardize the input data; A window-based multi-head self-attention module, configured to divide all pixels of corresponding input image into multiple small windows, and to calculate correlation between each of the pixels and the other pixels within each of the windows to capture local features; A residual connection and layer normalization module, configured to directly add input features to self-attention processing results from the previous module to form the residual connection, and then to perform layer normalization processing; A multilayer perceptron, configured to perform complex transformation and combination of the features through fully connected layers and nonlinear activation functions to enhance the feature representation capability of a model; A shifted window-based multi-head self-attention module, configured to divide all the pixels of the corresponding input image into the multiple windows, to shift the divided windows rightwards and downwards by the several pixels during division, and to calculate the correlation between each of the pixels and the other pixels within each of the windows to achieve information interaction between the different windows; An another residual connection and layer normalization module, configured to directly add the input features to the self-attention processing results from the previous module to form the residual connection, and then to perform layer normalization processing; An another multilayer perceptron, configured to perform complex transformation and combination of the features through the fully connected layers and the nonlinear activation functions to enhance the feature representation capability of the model; In some of the embodiments, the dual cross-attention module comprises: A first cross-attention mechanism layer, configured to take features of each of patches extracted by a SwinT module as keys and values, to take current classification label information as queries, to perform multi-head attention mechanism processing, to merge processing results, and then to perform layer normalization processing to obtain output results of the first cross-attention mechanism; A second cross-attention mechanism layer, configured to take the features of each of the patches extracted by the SwinT module as the keys and the values, to take the output results of the first cross-attention mechanism as queries, to perform multi-head attention mechanism processing, to merge the processing results, and then to perform layer normalization processing to obtain output results of the second cross-attention mechanism. In some of the embodiments, the label generation module comprises: The multilayer perceptron, configured to process the input data through the two fully connected layers to enhance the feature representation capability; A classification module, configured to calculate confidence via a softmax function to achieve classification. Compared with the prior art, the present invention has at least the following beneficial effects: In the embodiments of the present invention, the provided CA-SwinT-based hierarchical image classification method has better feature representation capability. This advantage primarily stems from the local window attention mechanism and window shifting strategy of the SwinT module, which, while reducing model 7 parameters and building a lightweight framework, better captures global and detailed information of the image; In the embodiments of the present invention, the provided CA-SwinT-based hierarchical image classification method has higher accuracy. The advantage primarily comes from the dual cross-attention mechanism, which enables effective information exchange between different modal features, enhancing the mapping capability between global / locaI image features and label semantics. Such mechanism helps to more accurately characterize the mapping between complex image content and semantics, increasing the recognition accuracy of the model; In the embodiments of the present invention, the provided CA-SwinT-based hierarchical image classification method has higher efficiency. The advantage primarily comes from the combination of the SwinT module and the dual cross-attention module, i.e., the entire CA-SwinT model; Wherein, the SwinT module, through the shifted window mechanism, divides the image into small windows and the attention mechanism is computed within local regions, which greatly reduces computational complexity. The shifted window mechanism ensures that features across windows can share information, further improving model efficiency; The dual cross-attention module effectively reduces redundant computation by exchanging feature attention across modalities, focusing only on important feature relationship mappings, which thereby improves computational efficiency. The CA-SwinT-based hierarchical image classification system of the present invention can implement the step of the described CA-SwinT-based hierarchical image classification method, and thus possesses at least all the features and advantages of the described CA-SwinT-based hierarchical image classification method, which will not be repeated here. Additional aspects and advantages of the present invention will be partly given in the following description, partly become apparent from the following description, or be learned through practice of the present invention. Description of drawings Figure 1 is a flowchart of a CA-SwinT-based hierarchical image classification method 8 disclosed in one embodiment of the present invention; Figure 2 is a schematic structural diagram of modules configured in the CA-SwinT-based hierarchical image classification method disclosed in one embodiment of the present invention. Embodiments Hereinafter, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. It is apparent that the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention. Hereinafter, the embodiments of the present invention are described in detail through specific embodiments and application scenarios in conjunction with the accompanying drawings. In some embodiments of the present invention, a CA-SwinT-based hierarchical image classification method is provided, wherein the method introduces main mechanisms of SwinTransformer and a dual cross-attention mechanism. A SwinT module, based on the SwinTransformer, effectively reduces model parameters, lowers model complexity, and extracts rich, fine-grained feature information from images. Simultaneously, the dual cross-attention mechanism can better construct a more comprehensive feature mapping relationship between images and labels, which thereby efficiently identifies multi-level labels of images. With reference to Figure 1, in some embodiments of the present invention, the method comprises the following three main parts, a SwinT module, a dual cross-attention module, and a label generation module. Wherein, the SwinT module is primarily responsible for extracting image features, the dual cross-attention module is responsible for modeling the relationship between the image and labels, and the label generation module is configured to obtain the final label information of 9 the image. The steps of the classification method comprises: Step 1, First, preprocess the image data and establish a label file (JSON file) corresponding to the image data, configured for inputting label information during model training and calculating model accuracy during model optimization. Step 2, The image data is first processed by the SwinT module, which is primarily responsible for extracting image features. The module structure is shown in Figure 2, mainly comprising patch partitioning, linear embedding, Swin modules, and patch merging modules, wherein the structure comprises sequentially arranged: a patch partitioning layer, a linear embedding layer, a two-level Swin module, a patch merging layer, an another two-level Swin module, and a linear embedding layer. Wherein: The patch partitioning layer, divides the image into a series of small, non-overlapping patches, which facilitates local processing of each patch by the model and effectively reduces the computational complexity of the model. Herein, the input image (HxWx3) is split into several 4x4 patches, for a total of (H / 4)x(W / 4)x48 patches, wherein 48 is the number of channels. The linear embedding layer, wherein each patch is flattened and processed through a fully connected linear embedding layer, transforming the pixel values of each patch into a higher-dimensional feature vector, to better capture features, the number of channels is adjusted to 96. The Swin module, originates from the Swin Transformer Block in the Swin Transformer model, wherein the module, through a series of carefully designed sub-modules, achieves feature perception and information fusion from local to global. The structure of the Swin module can be referred to in Figure 2, mainly comprising sequentially arranged: a layer normalization module, a window-based multi-head self-attention module (W-MSA), a residual connection and layer normalization module, a multilayer perceptron, a shifted window-based multi-head self-attention module (SW-MSA), an another residual connection and layer normalization module, and an another multilayer perceptron. In some embodiments of the present application, the role of layer normalization is to io facilitate neural network training and accelerate convergence by standardizing the input of each layer, enabling the model to achieve better performance with fewer training epochs; The implementation typically involves normalizing the activation values of each sample to have a mean of 0 and a variance of 1, then applying learnable scaling and shifting parameters. The window-based multi-head self-attention module (W-MSA) divides all pixels (i.e., the input image) into multiple fixed-size windows, then calculates the correlation between each pixel and other pixels within each window to capture local features. The shifted window-based multi-head self-attention module (SW-MSA) shifts the divided windows rightwards and downwards by X pixels during window division, which thereby achieves information interaction between different windows, which can significantly reduce computational load. The alternating use of W-MSA and SW-MSA allows the self-attention modules to gradually increase the perceptual range layer by layer, from local to global. Residual connection, directly adds the input features to the output of the self-attention module to form a residual connection, configured to reduce gradient vanishing, promote gradient flow, prevent degradation, and facilitate training of the model, which thereby optimizes the deep model; Combined with layer normalization, the training process can be further stabilized. Multilayer perceptron (MLP), performs complex transformation and combination of features through fully connected layers and nonlinear activation functions to enhance the feature representation capability of the model. The attention mechanism algorithm for each Swin module is as follows: >' = Qw-^ UMr )) + (1) r,=MLP(LN{r,))^ = Qsw-msa )) + rz = MLP(LN(rM )) + (4) In the aforementioned Swin module of the present invention, by alternately utilizing W-MSA and SW-MSA, the Swin module can gradually increase the perceptual range layer by layer, from capturing local features to global features; through window 11 partitioning and shifting operations, the Swin module can significantly reduce computational load and improve model efficiency; the use of layer normalization and residual connections makes the Swin module more stable during training and easier to optimize. Patch merging layer, primarily performs down-sampling on the image to reduce the size of the feature map and increase the number of channels. In the present embodiment, the height and width are halved, and the channels are doubled. Herein, the image size are changed into (H / 8)x(W / 8)xl92. Afterwards, it undergoes two more Swin modules and a linear transformation processing. After processing through the entire SwinT module, the obtained feature representation is: The size of the obtained image feature is (H / 8)x(W / 8)x^». In this embodiment, performing the two-stage Swin processing after the patch merging layer aims to further capture high-level features in the image, providing a more effective feature representation for subsequent classification. Performing another linear embedding processing is mainly for changing the number of the channels. Step 3, Introduce the dual cross-attention mechanism, utilizing the text label information input stream to guide the features of the image input stream, achieving full integration of information and constructing a fine-grained mapping between the image and hierarchical labels. With reference to Figure 2, in the dual cross-attention module, the feature vector r obtained from the SwinT module processing is configured as the Key and Value extracted from the image, and the corresponding label w is configured as the Query. Let , anc| be t^e learned weight matrices, then , Oi = WiQr Oi =WiQr J1 , ■ . The attention weight matrix is calculated based on the Query Q and Key K, the similarity between the query and key measured through a dot product operation, which is as follows: Attention3 (0 ,K3 ,V3 ) = soft max(^=^)E3 ^d3 Herein, j takes the value 1 or 2, representing mechanism or the second layer cross-attention (6) the first layer cross-attention mechanism, respectively. The O' / (-1 yi dimensions of , 1 , ' are all m, wherein 1 represents the number of dj attention heads, and * is the dimension of the key, configured to scale the dot . d3 = d3 / 1 product result. The total number of attention heads is 1, and k m . The mechanism allows each attention head to independently learn the correlation between global image features and current labels to generate image-label attention maps at different dimensional levels. Hx = LayerNorm(JVhxFconcat {Attentionf^,..., Attention^ ) + w) Subsequently, the outputs of the multi-head attention are concatenated and transformed by the linear transformation matrix Whi to generate the attention output. The output is reintegrated with the initial input via a residual connection and undergoes layer normalization to obtain the output H, of the first cross-attention mechanism. The present step helps mitigate overfitting and enhance the stability of the model. In the second layer of the dual cross-attention mechanism, the output H\ from the first layer is configured as the Query for further inquiry, while the original image features are reused as the Key and Value to further learn the mapping relationship between image features and hierarchical labels at a deeper level. For the attention of each head, a new set of linear transformation matrices is configured to calculate the attention scores of these inputs, which are then scaled and normalized to determine the weight distribution, see formula (6), wherein j takes the value 2. Similar to the first layer, the attention outputs of all heads are concatenated and then transformed by a linear matrix Wh2 to generate the final output of the present layer: ^2 = Wh2Fconcat(Attentio^ H2 represents the relational mapping representation obtained after comprehensively considering the association between the image and hierarchical labels, providing information-rich fused features for subsequent hierarchical classification. The output of the second layer cross-attention mechanism is combined with the output Hl of the first cross-attention mechanism via a residual connection. Subsequently, layer normalization is applied to the data to generate the final output Zx of the dual cross-attention mechanism, which is calculated as follows: Zx IxyerNornii H ~II ) The dual cross-attention mechanism is adopted to establish bidirectional cross-attention between the image and text labels. On one hand, text label information is configured to guide the learning of image features, such that the image features are more aligned with the description of the text labels; on the other hand, image features can also be configured to enrich the representation of text labels, such that they are more specific and accurate. The bidirectional flow of information helps achieve full integration between the image and text labels, constructing a more fine-grained mapping relationship between the image and hierarchical labels. Step 4, After obtaining the comprehensive attention information Zx, the label generation module is finally configured to obtain the hierarchical label information. The label generation module comprises an MLP layer and a softmax layer. The MLP layer utilizes two fully connected layers, followed by the softmax function to obtain the confidence score y for each label. y = soft max(MLP(Zx, w)) Finally, the model training adopts cross-entropy as the loss function. The formula is as follows: 1 N M L =--— X X Zik Zik N ,=i k=] -t \ Wherein, N is the total number of data samples, and M is the total number of label categories. y is the value of the true label for the current sample, and yjk is the value of the predicted label for the current sample. Parts of the present invention not described in detail can be understood with reference to prior art in the field or techniques known to those skilled in the art. The embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the specific implementations described above. The specific implementations described above are merely illustrative and not restrictive. Inspired by the present invention, those of ordinary skill in the art can make many forms without departing from the purpose of the present invention and the scope protected by the claims, all of which fall within the protection of the present invention.
Claims
1. A CA-SwinT-based hierarchical image classification method, comprising:SI, acquiring an original image and performing preprocessing on the original image;S2, extracting features from the image via a SwinT module;S3, fully integrating the extracted features and classification label information by a dual cross-attention mechanism, and constructing a fine-grained mapping between the image and hierarchical labels;S4, generating corresponding labels through a label generation module based on the fine-grained mapping from the step S3, and updating the classification label information;wherein the step of extracting features from the image via a SwinT module in the step S2 comprises:S201, patch partitioning processing, comprises dividing the image into a series of small patches;S202, linear embedding processing, comprises flattening each of the patches and processing it through a fully connected linear embedding layer to transform a pixel value of each of the patches into a high-dimensional feature vector;S203, two-level Swin module processing, comprises sequentially inputting processing results from the S202 into the two-level Swin module for feature extraction;S204, patch merging processing, comprises performing down-sampling on the image to half height and width thereof and to double channels thereof;S205, two-level Swin processing, comprises sequentially inputting processing results from the S204 into the two-level Swin module for feature extraction;S206, linear embedding processing, comprises applying a linear transformation to processing results from the S205.wherein the processing procedure of each of the stages of the Swin module in the step S203 and step S205 comprises the following steps that are performed sequentially:layer normalization processing, comprises standardizing data;window-based multi-head self-attention processing, comprises dividing all pixels of the input image into multiple windows, calculating correlation between each of the pixels and the other pixels within each of the windows to capture local features;residual connection processing, followed by layer normalization processing on the results, wherein the residual connection processing comprises directly adding the input features to self-attention processing results from the previous step, forming a residual connection;multilayer perceptron processing, comprises performing complex transformation and combination of the features through fully connected layers and nonlinear activation functions to enhance feature representation capability of a model;shifted window-based multi-head self-attention processing, comprises dividing all the pixels of the input image into the multiple windows, while shifting the divided windows rightwards and downwards by the several pixels during division, and calculating the correlation between each of the pixels and the other pixels within each of the windows to achieve information interaction between the different windows;residual connection processing, followed by layer normalization processing on the results, wherein the residual connection processing comprises directly adding the input features to the self-attention processing results from the previous step, forming the residual connection;multilayer perceptron processing, comprises performing complex transformation and combination of the features through the fully connected layers and the nonlinear activation functions to enhance the feature representation capability of the model;wherein the step of fully integrating the extracted features and classification label information by a dual cross-attention mechanism, and constructing a fine-grained mapping between the image and hierarchical labels in S3 comprises: S301, taking the features of each of the patches extracted by the SwinT module17as keys and values, and taking the current classification label information as queries, performing multi-head attention mechanism processing, merging the processing results, and then performing layer normalization processing to obtain output results of the first cross-attention mechanism;S302, taking the features of each of the patches extracted by the SwinT module as the keys and the values, and taking the current classification label information as queries, performing multi-head attention mechanism processing, merging the processing results, and then performing layer normalization processing to obtain output results of the second cross-attention mechanism;wherein the step of generating the corresponding labels through the label generation module comprises:S401, multilayer perceptron processing, comprises processing the input data through the two fully connected layers to enhance the feature representation capability; andS402, classification processing, comprises calculating confidence via a softmax function to achieve classification.
2. A CA-SwinT-based hierarchical image classification system, configured to implement the CA-SwinT-based hierarchical image classification method according to claim 1; wherein the system comprises:a preprocessing module, configured to receive the original image and perform preprocessing on the original image;a feature extraction module, configured to perform feature extraction on the image via the SwinT module;a dual cross-attention module, configured to fully integrate the extracted features and the classification label information by utilizing the dual cross-attention mechanism, and to construct the fine-grained mapping between the image and the hierarchical labels; anda label generation module, configured to generate the corresponding labels based on the fine-grained mapping obtained by the dual cross-attention module, and to update the classification label information.
183. The CA-SwinT-based hierarchical image classification method according to claim 2, wherein the feature extraction module comprises, connected sequentially:a patch partitioning layer, configured to divide the image into a series of small patches;a linear embedding layer, configured to flatten each of the patches and process it through a fully connected linear embedding layer to transform a pixel value of each of the patches into a high-dimensional feature vector;a two-level Swin module, configured to input processing results from the preceding linear embedding layer for feature extraction;a patch merging module, configured to halve height and width of the image and double channels thereof through down-sampling;an another two-level Swin module, configured to perform feature extraction on processing results from the preceding patch merging module; anda linear embedding module, configured to apply a linear transformation to processing results from the preceding two-level Swin module.
4. The CA-SwinT-based hierarchical image classification method according to claim3, wherein each level of the Swin module comprises, connected sequentially:a layer normalization layer, configured to standardize the input data;a window-based multi-head self-attention module, configured to divide all pixels of corresponding input image into multiple small windows, and to calculate correlation between each of the pixels and the other pixels within each of the windows to capture local features;a residual connection and layer normalization module, configured to directly add input features to self-attention processing results from the previous module to form the residual connection, and then to perform layer normalization processing to alleviate a gradient vanishing problem;a multilayer perceptron, configured to perform complex transformation and combination of the features through fully connected layers and nonlinear activation functions to enhance the feature representation capability of a model; a shifted window-based multi-head self-attention module, configured to divide19all the pixels of the corresponding input image into the multiple windows, to shift the divided windows rightwards and downwards by the several pixels during division, and to calculate the correlation between each of the pixels and the other pixels within each of the windows to achieve information interaction between the different windows;an another residual connection and layer normalization module, configured to directly add the input features to the self-attention processing results from the previous module to form the residual connection, and then to perform layer normalization processing to alleviate the gradient vanishing problem; andan another multilayer perceptron, configured to perform complex transformation and combination of the features through the fully connected layers and the nonlinear activation functions to enhance the feature representation capability of the model;5. The CA-SwinT-based hierarchical image classification system according to claim 4, wherein the dual cross-attention module comprises:a first cross-attention mechanism layer, configured to take features of each of patches extracted by a SwinT module as keys and values, to take current classification label information as queries, to perform multi-head attention mechanism processing, to merge processing results, and then to perform layer normalization processing to obtain output results of the first cross-attention mechanism;a second cross-attention mechanism layer, configured to take the features of each of the patches extracted by the SwinT module as the keys and the values, to take the output results of the first cross-attention mechanism as queries, to perform multi-head attention mechanism processing, to merge the processing results, and then to perform layer normalization processing to obtain output results of the second cross-attention mechanism.
6. The CA-SwinT-based hierarchical image classification system according to claim 4, wherein the label generation module comprises:the multilayer perceptron, configured to process the input data through the two20fully connected layers to enhance the feature representation capability;a classification module, configured to calculate confidence via a softmax function to achieve classification.
Citation Information
Patent Citations
Remote sensing image classification system and method based on multi-head self-attention module
CN113344070A
Lung CT image classification system based on domain knowledge and parallel separable convolution Swin Transform
CN117058448A
Multi-label image classification method based on deep learning
CN118447324A
Hyperspectral image classification method based on high-order interactive convolutional network
CN118506112A
Hierarchical image classification method and system based on CA-SwinT
CN119380108A
