A Zero-Shot Image Classification Method and System Based on Multi-Stage Modulation and Dynamic Fusion
The zero-shot image classification method based on multi-stage modulation and dynamic fusion solves the problems of feature loss caused by single-stage alignment and insufficient multi-level feature fusion in existing technologies, and achieves high-precision classification in complex scenarios, which is suitable for Martian ground feature monitoring and disaster change detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT SPACE SCI CENT CAS
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-31
AI Technical Summary
Existing zero-shot image classification methods suffer from feature loss due to single-stage image-text alignment in complex scenarios, lack the ability to dynamically adjust through multi-level feature fusion, and are difficult to effectively identify unseen categories.
A multi-stage modulation and dynamic fusion approach is adopted, which utilizes a state space model to achieve adaptive weight allocation of multi-stage aligned features through a visual coding module, a semantic coding module, a feature alignment module, and a multi-level feature fusion module, thereby improving discriminative ability and generalization performance.
It significantly improves zero-shot classification accuracy, enhances the model's classification performance in complex environments, and meets the low power consumption and high real-time requirements of deep space exploration equipment.
Smart Images
Figure CN122493099A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and specifically relates to a zero-shot image classification method and system that utilizes a state-space model for multi-level feature modulation and dynamic weight fusion, which can be used for planetary image perception, environmental monitoring, urban planning and natural disaster management. Background Technology
[0002] Zero-shot learning is a learning paradigm that leverages existing knowledge to identify unseen categories during the training phase. It establishes semantic relationships between seen and unseen categories by introducing auxiliary information such as attribute information, text descriptions, or semantic embeddings, thus overcoming the dependence of traditional supervised learning on labeled samples. As one of the fundamental tasks of zero-shot learning in computer vision, zero-shot image classification achieves the ability to identify unseen categories under the condition that the categories in the training set and the test set do not overlap, providing crucial technical support for various complex applications such as remote sensing image analysis and natural scene recognition.
[0003] Currently, zero-shot image classification methods have been widely studied in fields such as terrain and landform scene classification and flora and fauna category recognition, and are gradually expanding to applications such as planetary image perception, environmental monitoring, urban planning, and natural disaster management. In these applications, image data acquisition is costly, annotation is difficult, and the actual focus is often on new categories not encountered during the training phase. For example, in planetary image perception tasks, the available remote sensing image data is limited, while unseen categories often contain potential scientific value; in environmental monitoring and disaster management, natural changes or sudden events may create new land cover types or visual features, placing higher demands on the model's generalization ability. Therefore, designing zero-shot image classification methods capable of stably identifying unseen categories has significant practical implications and application value.
[0004] Existing zero-shot classification frameworks based on state-space models, such as Chinese patent CN120339693A, primarily focus on leveraging the linear sequence modeling capabilities of state-space models to achieve efficient interaction between image visual features and text semantic features when handling cross-modal information interaction, effectively reducing the computational complexity of the model. However, in practical applications, especially in extremely complex environments such as Martian geological exploration, this method still has the following significant limitations: First, there's the issue of feature loss due to single-stage image-text alignment. In complex scene images, such as those of Mars, distinguishing between "igneous rocks" and "sedimentary rocks" requires not only deep discriminative features (overall structure) but also shallow edge textures (delayed bedding) and detailed geometric contours in the middle layers. Existing techniques typically perform a single-stage image-text feature alignment at the end of the visual and semantic feature extractors. This "whole-to-whole" mapping often loses crucial low-level visual features, resulting in insufficient discriminative power for fine-grained categories.
[0005] Secondly, there is the limitation of single-stage image-text alignment. Existing technologies primarily focus on the equivalence of global image representation and macroscopic semantics in the alignment process, neglecting the hierarchical correspondence between visual hierarchy and semantic granularity. For example, "layer texture" in text corresponds to shallow visual features, while "sedimentary rock" corresponds to deep features. This non-hierarchical alignment pattern makes it difficult to construct a robust semantic projection space when dealing with highly abstract zero-sample category descriptions, thus limiting the model's generalization performance on unseen categories.
[0006] Furthermore, the multi-level feature fusion process lacks a dynamic weight selection mechanism. After introducing multi-stage alignment, efficiently integrating alignment features from different levels becomes a new challenge. Traditional fusion methods often employ simple cascading or static weighting, failing to dynamically adjust the contribution of each level of alignment features based on the specific content of the input image. For example, in complex scene exploration missions like Mars, the reliability of features at different stages can change dramatically in real time due to factors such as lighting, dust, and shooting angle. Fusion mechanisms lacking dynamic adjustment capabilities struggle to maintain stable classification performance in complex and variable scenarios, hindering their widespread adoption in practical applications.
[0007] In summary, existing technologies still have the following shortcomings in zero-shot image classification: (1) The lack of a mechanism to introduce semantic information at multiple visual levels results in the ineffective use of low-level discriminative features; (2) It lacks the ability to adaptively and dynamically fuse multi-level features, making it difficult to dynamically adjust feature contributions according to changes in input.
[0008] Therefore, there is an urgent need for a zero-shot image classification method that can introduce semantic constraints into multi-level features and achieve adaptive fusion, so as to improve the classification performance and stability of the model under the condition of no class seen. Summary of the Invention
[0009] The purpose of this invention is to solve the following problems existing in current image classification methods: (1) Existing methods usually perform single-stage visual and semantic alignment only at the end of feature extraction, resulting in a lack of effective correlation between shallow texture information and semantic information, which restricts the model's discriminative ability; (2) Existing multi-feature fusion methods mostly use fixed weighting or simple splicing, which lack the ability to dynamically adjust feature contributions based on input data, thus limiting the generalization performance of the model.
[0010] To address the aforementioned issues, this invention proposes a zero-shot image classification method and system based on multi-stage modulation and dynamic fusion. By constructing a multi-level image-text modulation space and utilizing the selective scanning mechanism of the state-space model to achieve adaptive weight allocation of multi-stage alignment results, the zero-shot classification accuracy in complex Martian scenarios is significantly improved.
[0011] To achieve the above objectives, the present invention proposes the following technical solution: A zero-shot image classification method based on multi-stage modulation and dynamic fusion, characterized by the following steps: Step S1: Obtain the dataset and divide it into a training sample set and a test sample set; Step S2, construct a zero-shot image classification model: An end-to-end training approach is used to construct a zero-shot image classification model, which includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. in, The visual encoding module is used to perform feature modeling on the input image in at least two stages and output visual features in at least two stages. The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct the multi-stage alignment features into a sequence according to the stage order, and to model the sequence based on the state space model. The fusion weights of the alignment features at each stage are adaptively generated through the state evolution process to obtain the final fused alignment features. The classification module is used to output the classification result of the image based on the fused alignment features; Step S3: Train the zero-shot image classification model using training samples of the visible categories; Step S4: Input the test samples of unseen categories into the trained zero-shot image classification network model, and output the classification results of unseen categories.
[0012] As one aspect of the above method, the dataset samples include image data, category semantic text description letters, and category labels; the dataset is randomly divided into training samples of visible categories and test samples of unseen categories.
[0013] As another aspect of the above method, the visual encoding module extracts multi-stage visual features with spatial resolution differences through a multi-level feature extraction network based on a state-space model, a convolutional neural network, or a Transformer network.
[0014] As another aspect of the above method, the feature alignment module's step of interacting with visual and semantic features at each stage includes: Step S2-1, Visual Feature Dimension Mapping: Perform dimension-consistent mapping on visual features at different stages.
[0015] Step S2-2, Visual semantic feature alignment based on multi-stage modulation: utilizing semantic features Extracted from each stage of the visual encoding module Visual features undergo constraint modulation at each alignment stage, generating staged alignment features through the hierarchical correlation between visual and semantic features. .
[0016] As another embodiment of the above method, the step of the multi-level feature fusion module adaptively generating fusion weights for each stage of the alignment features through a state evolution process to obtain the final fused alignment features includes: Visual semantic features that align modulation across multiple stages Arranged in sequence according to stage order, the data is input into a multi-level feature fusion module based on a state-space model. The selective scanning mechanism of the state space is used to model the contribution of aligned features at different stages. (5) In the formula, This refers to the fusion alignment features after adaptive dynamic fusion. For adaptive fusion weights, i represents the stage.
[0017] In another embodiment of the above method, the adaptive fusion weights are formed by the following steps: Step R1: Align the features of each stage. Mapped to state-space parameter terms : (6) (7) (8) In the formula, This is a commonly used anisotropic smooth activation function, with a step size of [missing information]. Represents the sampling interval during the discretization process; This is the weight matrix; For bias terms; It is an identity matrix. This is the discretized state transition matrix; This is the discretized input projection matrix.
[0018] Step R2 captures the correlation between levels through the state evolution equation, the formula of which is as follows: (9) Step R3: Generate dynamic fusion weights : (10) in, It is the Sigmoid activation function. The state projection matrix, The input projection matrix.
[0019] As a further embodiment of the above method, the step S3 is characterized by employing an end-to-end training approach, specifically including the following steps: Step S3-1: Construct the overall loss function for this network model: Step S3-2: Initialize the parameters of the visual encoding module, semantic encoding module, feature alignment module, multi-level feature fusion module, and classification module in the zero-shot image classification network model; Step S3-3: Input the training samples of the visible categories and the semantic information of the categories into the zero-shot image classification network model. Use the gradient descent method to iteratively update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and the trained zero-shot image classification network model is obtained.
[0020] As a further aspect of the above method, the overall loss function is characterized by: , in This is a semantic vector loss function based on cross-entropy loss. For based on The regression-like loss function of the paradigm These are the weight parameters of the regression loss function; Among them, the class semantic vector loss function based on cross-entropy loss ,based on Paradigm-like regression loss function They are represented as follows: ; ; in, It is an image Text information, It is an image Dynamic fusion representation of multi-stage visual features. This indicates the category to which the image belongs. This represents the total number of categories of visible images in the dataset.
[0021] As a further embodiment of the above method, the gradient descent method is characterized by the following specific steps: Step S3-3-1: Set the initial learning rate value. Set the maximum number of iterations T, the batch size B, and initialize the network training parameters; Step S3-3-2: Divide the training samples into batches according to the batch size, and input them sequentially into the zero-sample image classification network for backpropagation to update the total loss function. The gradient, according to Forward propagation is performed in the descent direction to update the state space parameters. The model parameters are included; the above calculations are performed on each batch of input networks until the entire training set of samples has been traversed to complete one training cycle. Step S3-3-3 repeats the process of step S3-3-2 until the maximum number of iterations T is reached, resulting in a zero-shot image classification model that can automatically adjust feature contributions according to image category.
[0022] The present invention also provides a zero-shot image classification device based on multi-stage modulation and dynamic fusion, used to implement the zero-shot image classification method based on multi-stage modulation and dynamic fusion, characterized in that it includes: The dataset acquisition module is used to acquire the dataset and divide it into training sample set and test sample set; The zero-shot image classification model is designed using an end-to-end training approach. This model includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. in, The visual encoding module is used to perform feature modeling on the input image in at least two stages and output visual features in at least two stages. The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct the multi-stage alignment features into a sequence according to the stage order, and to model the sequence based on the state space model. The fusion weights of the alignment features at each stage are adaptively generated through the state evolution process to obtain the final fused alignment features. The classification module is used to output the classification result of the image based on the fused alignment features; The training module is used to train the zero-shot image classification model using training samples of the visible categories; The input / output module is used to input test samples of unseen categories into the trained zero-shot image classification network model and output the classification results of unseen categories.
[0023] The feature alignment module is a dual-branch feature alignment module comprising a mapping layer, a convolutional layer, an activation function, a state space model, a normalization and residual connection layer, and a Mamba-2 module cascaded together. One branch inputs the visual features of the stage, and the other branch inputs the semantic features. The semantic information is used as a guiding signal to perform nonlinear modulation on the visual features of multiple stages.
[0024] Compared with the prior art, the invention of this application has the following beneficial effects: (1) Improve the ability to distinguish categories.
[0025] This invention achieves multi-level interaction between visual features and semantic information, from local texture to global semantics, effectively improving the discriminative ability of zero-shot classification models. This invention constructs a unified framework including a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. Through the multi-stage feature alignment module, a cascaded correspondence between visual levels and semantic features is explicitly established. Compared to existing technologies that rely solely on end features for alignment, this invention can capture the correlation information between shallow edge textures and mid-level geometric contours and semantic information, fundamentally solving the problem of discriminative feature loss caused by single-stage alignment.
[0026] (2) Enhance the consistency between semantic and visual alignment.
[0027] In this invention, textual information is used as a modulation signal in the feature alignment module to semantically constrain visual features at each stage. This design can dynamically enhance the response of the target region based on semantic relevance, thereby obtaining more discriminative visual semantic alignment features and providing more accurate information for classifying unseen categories in zero-shot scenarios.
[0028] (3) Achieve dynamic fusion of multi-stage alignment features.
[0029] The multi-level feature fusion module designed in this invention effectively achieves adaptive allocation of category for multi-stage aligned feature fusion weights by fully utilizing the selective scanning characteristics of the state space model.
[0030] (4) Balancing performance and computational efficiency.
[0031] This invention improves zero-shot classification while maintaining good computational complexity and real-time performance. Benefiting from the linear complexity advantage of state-space models in long sequence modeling, this invention, despite introducing multi-stage feature alignment and dynamic fusion mechanisms, does not lead to an exponential increase in computational load. The end-to-end architecture design allows input images to directly and quickly obtain classification results through the network, meeting the stringent requirements of deep space exploration equipment for low power consumption, high real-time performance, and embedded deployment.
[0032] (5) The structure is uniform and easy to expand. This invention constructs a unified framework that includes multi-stage semantic modulation and dynamic fusion. The modules work together to avoid the information fragmentation problem caused by the independent design of modules in traditional methods. At the same time, the framework can be adapted to different types of visual coding or semantic coding models and has good scalability.
[0033] Compared with existing technologies, the technical solution of this invention can not only better mine the multi-level shared information between images and auxiliary semantics, but also significantly improve the classification performance of unseen categories in complex environments under zero-sample settings through hierarchical modulation and dynamic selection mechanisms, providing more robust technical support for high-precision space exploration missions such as Martian surface feature monitoring and disaster change detection. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the method flow of a specific embodiment of the present invention; Figure 2 This is a schematic diagram of the zero-shot image classification model structure constructed in a specific embodiment of the present invention; Figure 3 This is an example image of a Mars scene according to a specific embodiment of the present invention. Detailed Implementation
[0035] The following examples further illustrate the technical solutions and effects provided in this application.
[0036] like Figure 1 As shown in the figure, a zero-shot image classification method based on multi-stage modulation and dynamic fusion provided by a specific embodiment of the present invention includes the following steps: Step S1: Obtain the dataset and divide it into a training sample set and a test sample set; To obtain visible category image data and semantic information for all categories, the dataset samples include image data, category semantic text descriptions, and category labels. The dataset is randomly divided into training and testing samples, for example, according to a ratio of 6 / 4, 7 / 3, and 8 / 2 of the total samples: 6 visible categories / 4 unseen categories, 7 visible categories / 3 unseen categories, and 8 visible categories / 2 unseen categories. Here, "unseen categories" should be understood as categories not seen during the training phase. For example, with 6 visible categories / 4 unseen categories, the 6 visible categories are the input categories during the training phase, which the model has seen, while the 4 unseen categories are the categories to be identified during the testing phase, which the model has not seen. There is no overlap between the 6 visible categories and the 4 unseen categories. A detailed classification example is shown in Table 1.
[0037] Step S2, construct the zero-shot image classification model architecture: like Figure 2 As shown, an end-to-end training method is used to construct a zero-shot image classification model. This model includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. The text description information given in the figure is sample text input in English, but is not limited to any particular language.
[0038] in, The visual encoding module is used to perform feature modeling on the input image at multiple stages and output visual features at multiple stages; the multiple stages include at least two stages.
[0039] The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct a sequence of the multi-stage alignment features according to the stage order, and to model the sequence based on a state-space model. It adaptively generates the fusion weights of the alignment features at each stage through a state evolution process to obtain the final fused alignment feature. The input of the multi-level feature fusion module is the output of multiple feature alignment modules, which realize hierarchical modulation and alignment of multi-stage visual features. The multi-level feature fusion module is a network that adaptively adjusts the multi-level alignment features to generate the final fused alignment feature using a dynamic scanning mechanism based on state-space modeling. That is, alignment needs to be performed at multiple stages, each generating an alignment feature, and then these features need to be fused together into a single alignment feature. This process requires adaptive adjustment of the fusion, which will be detailed later.
[0040] The classification module is used to output the classification result of the image based on the fused alignment features.
[0041] Preferably, the visual encoding module extracts multi-stage visual features with spatial resolution differences based on a state-space model, convolutional neural network, or Transformer network through a multi-level feature extraction network. An example of multi-level visual feature extraction steps is as follows: Image The input is fed into the visual encoding module, where features are extracted using state-space modeling to obtain... The visual feature representations corresponding to each processing stage are given by the following formulas: (1) In the formula, Indicates the first Visual features output at each stage This refers to the number of stages.
[0042] The visual encoding module extracts a visual feature pyramid with spatial resolution differences through multi-level state space modeling. Those skilled in the art should understand that the number of layers in the multi-level state space modeling can differ at each stage, resulting in different feature dimensions and thus different focuses in the extracted information. For example, Figure 2 The diagram illustrates four stages of visual features, each outputting to a feature alignment module. Specifically, in this visual encoder, sample data is input to a convolutional layer, and the output of the convolutional layer is fed to stage 1, which includes two visual state space (VSS) modules. The output of stage 1 is then mapped and input to the feature alignment module. Furthermore, the output of stage 1 enters stage 2, which includes downsampling and two visual state space modules, and the output of stage 2 is mapped and input to the corresponding feature alignment module. Similarly, the output of stage 2 enters stage 3, which includes downsampling and nine visual state space modules, and the output of stage 3 is mapped and input to the corresponding feature alignment module. Finally, the output of stage 3 enters stage 4, which includes downsampling and two visual state space modules, and the output of stage 4 is sent to the corresponding feature alignment module.
[0043] The steps of the semantic encoding module in performing feature encoding of semantic information are as follows: According to the input format required by the semantic encoding module, the semantic information of the category is reconstructed into the corresponding input format.
[0044] The category text description or attribute information corresponding to the image is input into the semantic encoding module to obtain the semantic feature representation. The specific formula is as follows: (2) In the formula, This indicates the category description or attribute information input. This represents a semantic encoding function.
[0045] The steps by which the feature alignment module interacts with the visual and semantic features at each stage include: Step S2-1, visual feature dimension mapping; Visual features at different stages Semantic features of the corresponding levels Input feature alignment module; To achieve interaction between multi-level visual features and semantic features, the visual features extracted at each stage of the visual encoding module in step S2 are first subjected to dimensionality consistency mapping, as shown in the following formula: (3) In the formula, This represents a feature mapping function used to map visual features from different stages to a unified common space.
[0046] Step S2-2, visual semantic feature alignment based on multi-stage modulation: In each alignment stage, semantic features are utilized. visual features Perform constraint modulation to generate staged alignment features based on the hierarchical correlation between visual and semantic features. This achieves a progressive constraint from shallow texture alignment to deep semantic alignment. Specifically: In the In each visual feature processing stage, the feature alignment module based on the state space model utilizes the semantic feature representation extracted by the semantic encoding module. As a modulated signal, the mapped visual features To perform interactive alignment, the specific formula is as follows: (4) In the formula, It is an alignment function based on semantic feature representation modulation, which guides the enhancement of category-related features in visual features through semantic feature representation.
[0047] The feature alignment module is a dual-branch feature alignment module comprising a mapping layer, a convolutional layer, an activation function, a state-space model, a normalization and residual connection layer, and a Mamba-2 module cascaded together. Its goal is to utilize semantic information as a guiding signal to nonlinearly modulate visual features at multiple stages, generating aligned features with hierarchical semantic discriminative capabilities. Specific implementation details are as follows: Step AS1, Dimensionality Consistency Mapping: Due to the features output by the visual encoder at different stages... Given different spatial resolutions and channel dimensions, a linear mapping layer is first used to map the visual features at each stage to semantic features. In the same common dimensional space, preliminary aligned visual candidate features are obtained. .
[0048] Step AS2, semantically guided hierarchical visual feature modulation: First, the first branch of the feature alignment module modulates semantic features... As a modulation factor, it is input into the mapping layer and convolutional layer to obtain the semantic state space model parameters. , , and semantic modulation vector Similarly, the second branch inputs the visual features from each stage into the mapping layer and convolutional layer to obtain the parameters of the visual state space model. , , and the visual vector to be modulated ; Subsequently, the results from the two branches are input into the state-space model, which utilizes its selective scanning mechanism to perform spatial sequence scanning of visual features based on semantic modulation vectors. Dynamically adjust the visual vector to be modulated The parameters of the visual state space model are used, and the state is modulated by combining the semantic constraints of the current level.
[0049] Step AS3, Normalization and Residual Connection: To ensure the stability of training, the modulation results of each stage are processed by layer normalization and residual connections are introduced to effectively fuse the original visual information with the modulated visual information, thereby obtaining semantically guided multi-stage visual features.
[0050] Step AS4, Visual-Semantic Feature Alignment: The visual and semantic features modulated at each stage are combined into a sequence and input into the Mamba-2 module. By utilizing the asymmetric attention characteristics of Mamba-2, the semantic features are actively constrained to the visual features, generating an aligned representation with hierarchical consistency, and obtaining the final multi-stage visual-semantic alignment features.
[0051] The multi-level feature fusion module adaptively generates fusion weights for each stage of the alignment features through a state evolution process to obtain the final fused alignment features. Visual semantic features aligned with the modulation of multiple stages generated in step S2-2 Arranged in sequence according to stage order, the data is input into a multi-level feature fusion module based on a state-space model. The contribution of aligned features at different stages is modeled using a selective scanning mechanism of the state space, as shown in the following formula: (5) In the formula, This refers to the multi-level fusion alignment features after adaptive dynamic fusion. The adaptive fusion weights are used to dynamically adjust the contribution of features from different stages to the final classification features.
[0052] The multi-level feature fusion module uses a selective scanning mechanism based on a state-space model for dynamic weighting, and the specific formula is as follows: First, align the features of each stage. Mapped to state-space parameter terms : (6) (7) (8) In the formula, This is a commonly used anisotropic smooth activation function, with a step size of [missing information]. This represents the sampling interval during the discretization process; This is the weight matrix; For bias terms; It is an identity matrix. This is the discretized state transition matrix; This is the discretized input projection matrix. pass Nonlinear transformation, while ensuring parameters Under the premise of strict positive values, selective adjustment of the sampling frequency of feature signals at different levels is realized, thereby enhancing the model's ability to dynamically perceive multi-level information.
[0053] The correlation between levels is captured by the state evolution equation, and its formula is as follows: (9) Finally, dynamic fusion weights are generated. Achieve feature The weighted aggregation formula is as follows: (10) in, It is the Sigmoid activation function. The state projection matrix, The input projection matrix.
[0054] Step S3: Train the zero-shot image classification model; The training image is represented as The labels corresponding to the training images are The category semantic vector corresponding to the training image is ,in For the first i One training image, For the first i A tag, For the first i Each category semantic vector, i The range is from 1 to N , This indicates the number of training images.
[0055] An end-to-end training approach is adopted, and the training specifically includes the following steps: Step S3-1: Construct the overall loss function for this network model: ; in This is a semantic vector loss function based on cross-entropy loss. For based on The regression-like loss function of the paradigm These are the weight parameters of the regression loss function; Among them, the class semantic vector loss function based on cross-entropy loss ,based on Paradigm-like regression loss function They are represented as follows: ; ; in, It is an image Text information, It is an image Dynamic fusion representation of multi-stage visual features. This indicates the category to which the image belongs. This represents the total number of categories of visible images in the dataset.
[0056] Step S3-2: Initialize the parameters of the visual encoding module, semantic encoding module, feature alignment module, multi-level feature fusion module, and classification module in the zero-shot image classification network model; Step S3-3: Input the visible category training samples and category semantic information into the zero-shot image classification network model. Use the gradient descent method to iteratively update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and the trained zero-shot image classification network model is obtained. In this approach, a gradient descent method is employed, with differentiated learning rates used between the visual encoding module and the multi-level feature fusion module to balance the feature extraction capability of the backbone network with the dynamic selection capability of the fusion module. The parameters in the zero-shot image classification network are iteratively updated as follows: Step S3-3-1: Set the initial learning rate value. Set the maximum number of iterations T, the batch size B, and initialize the network training parameters; Step S3-3-2: Divide the training samples into batches according to the batch size, and input them sequentially into the zero-sample image classification network for backpropagation to update the total loss function. The gradient, according to Forward propagation is performed in the descent direction to update the state space parameters. The model parameters are included. The above calculations are performed on each batch of input network samples until the entire training set of samples has been traversed, completing one training cycle. Step S3-3-3: Repeat step S3-3-2 until the maximum number of iterations T is reached, resulting in a zero-shot image classification model that can automatically adjust feature contributions based on image category. Step S4: Input the unseen class test samples into the trained zero-shot image classification network model and output the unseen class image classification results.
[0057] During the execution of the above zero-shot image classification method, the multi-level features have undergone feature distribution changes through semantic modulation before entering the fusion module; the fusion weights are calculated based on the state variables of the state space model.
[0058] Those skilled in the art will understand that, according to the technical solution of the present invention, after the classification network model is trained, it can classify newly acquired images in an autonomous working state. Specifically, the model can classify newly acquired input images based on the stored category semantic text description information and category labels (in one possible implementation, such as deep space exploration: the category semantic text description information and category labels have been obtained through other means, but the images have not yet been obtained). The classification result is output in the form of probabilities corresponding to each category label. As described below, the output result is the probability of the label to which the image sample belongs. Therefore, the unseen category test sample in step S4 should be understood to include the above-mentioned situation.
[0059] In other embodiments, the multi-level feature fusion module may be implemented using other sequence modeling structures, including but not limited to recurrent neural networks or other dynamic modeling models.
[0060] This invention specifically uses the zero-shot Mars scene classification standard dataset ZS-Mars as an example for classification. The images in this dataset and their corresponding ground truth labels are as follows: Figure 3 As shown, it includes 10 types of scenarios.
[0061] Reference Figure 1 The specific implementation steps of this example are as follows: Step 1: Randomly divide the training and test samples.
[0062] 1.1) For the ZSMars dataset, the training (visible class) and test (unseen class) samples are divided into training (visible class) and test (unseen class) samples according to the ratio of 6 / 4, 7 / 3, 8 / 2, that is, 6 visible class / 4 unseen class, 7 visible class / 3 unseen class, 8 visible class / 2 unseen class; 1.2) Label each training sample as follows: Formal encoding as a vector Each test sample is labeled with Formal encoding as a vector .
[0063] Step 2: Build a visual encoder based on a state-space model.
[0064] 2.1) Image features are extracted using a bidirectional scanning method. A visual encoder VMamba based on Mamba is built, with an input image size of 224×224×3. 2.2) From Figure 3 The samples shown are divided into training set and test set samples and their corresponding labels according to step 1. The training set is represented as follows: , tag as , For any training sample, for different visible / invisible class ratios, the subscript... The product of the number of visible categories and the number of scenes in each category. Corresponding tags The test set is represented as , tag as , For any test sample, for different visible / invisible category ratios, the subscript... The product of the number of invisible categories and the number of scenes in each category, Corresponding tags ; Step 3: Build a semantic encoder based on a state-space model.
[0065] 3.1) Extract semantic features from the text using a selective scanning mechanism and build a semantic encoder Mamba based on a state space model.
[0066] from Figure 3 The sample labels shown are obtained by dividing the training set and the test set according to step 1. Specifically, for different visible / invisible category ratios, the number of training samples corresponds to the product of the number of visible categories and the number of scenes in each category, while the number of test samples corresponds to the product of the number of invisible categories and the number of scenes in each category. The remaining sample labels are used as the test set. Step 4: Construct a feature alignment module based on a state-space model.
[0067] A dual-branch feature alignment module is constructed, consisting of a mapping layer, convolutional layer, activation function, state-space model, and normalization layer. The goal is to utilize semantic information as a guiding signal to nonlinearly modulate visual features across multiple stages, generating aligned features with hierarchical semantic discriminative capabilities. Specific implementation details are as follows: 4.1) Dimensionality Consistency Mapping: Due to the different features output by the visual encoder at different stages... Given different spatial resolutions and channel dimensions, a linear mapping layer is first used to map the visual features at each stage to semantic features. In the same common dimensional space, preliminary aligned visual candidate features are obtained. .
[0068] 4.2) Semantically Guided Hierarchical Visual Feature Modulation: First, the first branch of the feature alignment module modulates semantic features... As a modulation factor, it is input into the mapping layer and convolutional layer to obtain the semantic state space model parameters. , , and semantic modulation vector Similarly, the second branch maps the visual features from each stage to the mapping layer and convolutional layer to obtain the parameters of the visual state space model. , , and the visual vector to be modulated ; Subsequently, the results from the two branches are input into the state-space model, which utilizes its selective scanning mechanism to perform spatial sequence scanning of visual features based on semantic modulation vectors. Dynamically adjust the visual vector to be modulated The parameters of the visual state space model are used, and the state is modulated by combining the semantic constraints of the current level.
[0069] 4.3) Normalization and residual connection: In order to ensure the stability of training, the modulation results of each stage are processed by layer normalization and residual connection is introduced to effectively fuse the original visual information with the modulated visual information to obtain semantically guided multi-stage visual features.
[0070] 4.4) Visual-semantic feature alignment: The visual and semantic features modulated at each stage are combined into a sequence and input into the Mamba-2 module. By utilizing the asymmetric attention characteristics of Mamba-2, the semantic features are actively constrained to the visual features, generating an aligned representation with hierarchical consistency, and finally obtaining the multi-stage visual-semantic alignment features.
[0071] Step 5: Construct a multi-level feature fusion module based on a state-space model.
[0072] Unlike traditional weighted fusion, the parameters of this invention... (Step size) is influenced by input features Real-time guidance. When the image and text alignment quality is high at a certain stage (e.g., the texture details of Martian rocks are clear), Increasing the value of the feature at that level enhances the response of the state-space model at that moment, thereby adaptively increasing the contribution of that level of feature to the final classification feature.
[0073] Reference Figure 2 The specific implementation of this step is as follows: First, align the features of each stage. Mapped to state-space parameter terms : (14) (15) (16) In the formula, This is a commonly used anisotropic smooth activation function, with a step size of [missing information]. This represents the sampling interval during the discretization process. pass Nonlinear transformation, while ensuring parameters Under the premise of strict positive values, selective adjustment of the sampling frequency of feature signals at different levels is realized, thereby enhancing the model's ability to dynamically perceive multi-level information.
[0074] The correlation between levels is captured by the state evolution equation, and its formula is as follows: (17) Finally, dynamic fusion weights are generated. Achieve feature The weighted aggregation formula is as follows: (18) Step 6: Construct a zero-shot image classification model based on a state-space model.
[0075] 6.1) The visual encoder, semantic encoder, feature alignment module, and multi-level feature fusion module described above are sequentially cascaded to form a visual semantic feature extraction and alignment network; 6.2) Constructing a classifier . It includes one fully connected layer to transform the size of the transformation matrix, so as to convert the multi-level alignment features output by the above visual semantic feature extraction alignment network into probability vectors belonging to each category. This is used to determine the category to which a sample belongs.
[0076] Step 7: Construct the loss function for the zero-shot image classification model based on the state-space model.
[0077] 7.1) Constructing the hierarchical consistency loss function Used to force alignment features in the output of each stage. The semantic description corresponding to it in Euclidean space (or cosine space) To maintain consistency, the specific formula is as follows: (19) in, It is the first The visual vector output by semantic modulation at each stage These are the semantic features corresponding to the image; 7.2) Construct based on Paradigm-like regression loss function This is used to constrain the image representation to ensure that it is accurately mapped to the corresponding text embedding. The specific formula is as follows: (20) 7.3) Constructing the cross-entropy loss function This is used to force an image to have the highest compatibility score with its corresponding class semantic vector. The specific formula is as follows: (twenty one) in, This indicates the category to which the image belongs. This represents the total number of categories of visible images in the dataset; Step 8: Train the parameters of the zero-shot image classification model using training samples and the gradient descent method.
[0078] 8.1) Set the initial learning rate value Maximum number of iterations T, batch size B; 8.2) The parameters for initializing the zero-shot image classification network model include auxiliary parameters in the visual encoder VMamba. , , , Auxiliary parameters in the semantic encoder Mamba , , , Auxiliary parameters of the feature alignment module , , , , , , , Auxiliary parameters of the multi-level feature fusion module , , , and the parameters in the classifier ; 8.3) Divide the training samples into batches according to the batch size, and input them sequentially into the unbalanced hyperspectral image classification network for backpropagation to update the total loss function. The gradient, according to The network training parameters are updated by forward propagation in the descent direction. The above calculation is performed on each batch of input network until the entire training set samples are input network to complete one training cycle. 8.4) Repeat the process in 8.3) until the maximum number of iterations T is reached, and a trained zero-shot classification model is obtained.
[0079] Step 9: Use the trained zero-shot image classification model to classify the test samples.
[0080] Each test sample is input into the trained zero-shot image classification model to obtain the probability of the label belonging to each test sample. : ; in, For normalization function, For the total loss function, For test samples, The label corresponding to the c-th test sample. The semantic features corresponding to the c-th class of test samples, The total number of test samples, , , and These are auxiliary parameters in the VMamba visual encoder. , , and These are auxiliary parameters in the semantic encoder Mamba. , , , , , , and These are auxiliary parameters for the feature alignment module. , , and These are auxiliary parameters for the multi-level feature fusion module. These are the trainable parameters of the classifier.
[0081] The effects of the specific embodiments of the present invention can be further illustrated by the following simulation results.
[0082] I. Simulation Conditions 1. Test data: Using ZSMars data, semantic information is expressed as sentence vectors representing long sequence inputs. Images are tested according to three visible / invisible ratios: a 6 / 4 ratio (6 visible images for training, 4 unseen images for testing); a 7 / 3 ratio (7 visible images for training, 3 unseen images for testing); and an 8 / 2 ratio (8 visible images for training, 2 unseen images for testing). The specific visible / invisible category divisions are shown in Table 1.
[0083] 2. Test environment: It uses a Linux system, an NVIDIA GeForce GTX 4090 GPU, and the PyTorch deep learning framework.
[0084] Table 1. Classification of Visible / Invisible Categories
[0085] II. Simulation Content Simulation 1: Under the above conditions, the existing zero-shot image scene classification method UniMamba based on the state-space model was used to classify the ZSMars data. The results are shown in Table 2.
[0086] Simulation 2: Under the above conditions, the ZSMars data were classified using the method of this invention, and the results are shown in Table 2.
[0087] A comparison in Table 2 reveals that the method proposed in this invention achieves high classification accuracy.
[0088] Evaluation indicators The two methods were tested 10 times using the same random number, and the overall classification accuracy OA was calculated. The test results are shown in Table 1.
[0089] The formula for calculating the indicator is as follows: (twenty two) in, To determine the number of items correctly classified into this category, The number of samples of this class that were incorrectly classified into other classes. The number of non-class samples that are classified into other categories. This represents the number of other samples that were incorrectly classified into this class. Table 2 Simulation results of the present invention and the comparative method
[0090] As can be seen from Table 2, compared with the current advanced UniMamba method, the present invention can achieve higher classification accuracy under three visible / invisible ratios, especially under low visible / invisible ratios (6 / 4).
[0091] As another implementation, the present invention provides a zero-shot image classification device based on multi-stage modulation and dynamic fusion, for implementing the zero-shot image classification method based on multi-stage modulation and dynamic fusion, comprising: The dataset acquisition module is used to acquire the dataset and divide it into training sample set and test sample set; The zero-shot image classification model is designed using an end-to-end training approach. This model includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. in, The visual encoding module is used to perform feature modeling on the input image in at least two stages and output visual features in at least two stages. The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct the multi-stage alignment features into a sequence according to the stage order, and to model the sequence based on the state space model. The fusion weights of the alignment features at each stage are adaptively generated through the state evolution process to obtain the final fused alignment features. The classification module is used to output the classification result of the image based on the fused alignment features; The training module is used to train the zero-shot image classification model using training samples of the visible categories; The input / output module is used to input test samples of unseen categories into the trained zero-shot image classification network model and output the classification results of unseen categories. The feature alignment module is a dual-branch feature alignment module comprising a mapping layer, a convolutional layer, an activation function, a state space model, a normalization and residual connection layer, and a Mamba-2 module cascaded together. One branch inputs the visual features of the stage, and the other branch inputs the semantic features. The semantic information is used as a guiding signal to perform nonlinear modulation on the visual features of multiple stages.
[0092] As can be seen from the above detailed description of this application, the innovative points of this invention include the following: (1) Using text information as a modulation signal, semantic constraints are applied to visual features at each stage to achieve multi-stage visual semantic alignment. (2) To address the issue of inconsistent contributions of alignment features to category labels at different stages, a multi-level feature fusion module was designed to achieve adaptive modulation of the fusion weights of alignment features at different stages. (3) An end-to-end zero-shot classification model framework based on the state space model was designed.
[0093] Accordingly, the significant technical effects of this invention include the following: (1) Improve the ability to distinguish categories.
[0094] This invention achieves multi-level interaction between visual features and semantic information, from local texture to global semantics, effectively improving the discriminative ability of zero-shot classification models. This invention constructs an end-to-end framework comprising a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. Through multi-level feature modulation alignment, it explicitly establishes a cascaded correspondence between visual levels and semantic features. Compared to existing technologies that rely solely on end features for alignment, this invention can capture the correlation information between shallow edge textures and mid-level geometric contours and semantic information, fundamentally solving the problem of discriminative feature loss caused by single-stage alignment.
[0095] (2) Enhance the consistency between semantic and visual alignment.
[0096] In this invention, textual information is used as a modulation signal in the feature alignment module to semantically constrain visual features at each stage. This design can dynamically enhance the response of the target region based on semantic relevance, thereby obtaining more discriminative visual semantic alignment features and providing more accurate information for classifying unseen categories in zero-shot scenarios.
[0097] (3) Achieve dynamic fusion of multi-stage alignment features.
[0098] The multi-level feature fusion module designed in this invention effectively achieves adaptive allocation of category for multi-stage aligned feature fusion weights by fully utilizing the selective scanning characteristics of the state space model.
[0099] (4) Balancing performance and computational efficiency.
[0100] This invention improves zero-shot classification while maintaining good computational complexity and real-time performance. Benefiting from the linear complexity advantage of state-space models in long sequence modeling, this invention, despite introducing multi-stage feature alignment and dynamic fusion mechanisms, does not lead to an exponential increase in computational load. The end-to-end architecture design allows input images to directly and quickly obtain classification results through the network, meeting the stringent requirements of deep space exploration equipment for low power consumption, high real-time performance, and embedded deployment.
[0101] (5) The structure is uniform and easy to expand.
[0102] This invention constructs a unified framework that includes multi-stage semantic modulation and dynamic fusion. The modules work together to avoid the information fragmentation problem caused by the independent design of modules in traditional methods. At the same time, the framework can be adapted to different types of visual coding or semantic coding models and has good scalability.
[0103] It is worth noting that the technical solution of this invention is not a simple superposition of existing multi-level feature extraction methods and semantic alignment methods, but rather introduces a synergistic mechanism between "multi-level semantic modulation" and "dynamic fusion based on state evolution" in its structure, as explained in detail below: (1) In the feature extraction stage, semantic information is introduced at multiple levels to participate in the generation process of visual features at each level, thereby changing the representation form of the original visual features. (2) In the feature fusion stage, the multi-level features after semantic modulation are constructed into a sequence, and a unified model is used to model them, so that the fusion weight is adaptively generated by the dynamic relationship between the features; Therefore, in this invention, the distribution of features at each level has changed during the modulation stage, and the fusion process depends on the evolution of the modulated feature state, which is different from the processing method of "extracting features independently first and then performing fixed fusion" in the prior art.
[0104] Therefore, the technical solution of the present invention forms a holistic design that is dependent on the preceding and following parts, and cannot be obtained by simply combining the existing multi-layer feature method with the traditional fusion method.
[0105] In summary, the technical solution of this invention, through the combination structure of "multi-level semantic modulation + dynamic fusion based on state evolution", enables semantic information to participate in the entire feature extraction process and achieves adaptive allocation of feature contributions in the fusion stage, thus forming a technical path that is different from existing single-stage alignment and static fusion methods.
[0106] If any of the above steps are missing, and if multi-level semantic modulation is not performed, there will be a lack of semantic consistency between features at different levels, making it difficult to establish a stable category discrimination relationship through subsequent fusion. Without a state evolution-based fusion mechanism, it is impossible to dynamically adjust the contribution of features at each level according to changes in input. Therefore, the various technical features in the technical solution of the present invention are interdependent and work together to achieve the above-mentioned significant technical effects.
[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A zero-shot image classification method based on multi-stage modulation and dynamic fusion, characterized in that, Includes the following steps: Step S1: Obtain the dataset and divide it into a training sample set and a test sample set; Step S2, construct a zero-shot image classification model: An end-to-end training approach is used to construct a zero-shot image classification model, which includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. in, The visual encoding module is used to perform feature modeling on the input image in at least two stages and output visual features in at least two stages. The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct the multi-stage alignment features into a sequence according to the stage order, and to model the sequence based on the state space model. The fusion weights of the alignment features at each stage are adaptively generated through the state evolution process to obtain the final fused alignment features. The classification module is used to output the classification result of the image based on the fused alignment features; Step S3: Train the zero-shot image classification model using training samples of the visible categories; Step S4: Input the test samples of unseen categories into the trained zero-shot image classification network model, and output the classification results of unseen categories.
2. The zero-shot image classification method of claim 1, wherein, The dataset samples include image data, category semantic text description letters, and category labels; the dataset is randomly divided into training samples of visible categories and test samples of unseen categories.
3. The zero-shot image classification method according to claim 1, characterized in that, The visual encoding module extracts multi-stage visual features with spatial resolution differences based on a state-space model, convolutional neural network, or Transformer network through a multi-level feature extraction network.
4. The zero-shot image classification method according to claim 1, characterized in that, The steps by which the feature alignment module interacts with the visual and semantic features at each stage include: Step S2-1, Visual Feature Dimension Mapping; Perform dimension-consistent mapping on visual features at different stages; Step S2-2, Visual semantic feature alignment based on multi-stage modulation: utilizing semantic features Extracted from each stage of the visual encoding module Visual features undergo constraint modulation at each alignment stage, generating staged alignment features through the hierarchical correlation between visual and semantic features. .
5. The zero-shot image classification method according to claim 1, characterized in that, The multi-level feature fusion module adaptively generates fusion weights for each stage of the alignment features through a state evolution process to obtain the final fused alignment features. The steps include: Visual semantic features that align modulation across multiple stages Arranged in sequence according to stage order, the data is input into a multi-level feature fusion module based on a state-space model. The selective scanning mechanism of the state space is used to model the contribution of aligned features at different stages. (5) In the formula, This refers to the fusion alignment features after adaptive dynamic fusion. For adaptive fusion weights, i represents the stage.
6. The zero-shot image classification method according to claim 5, characterized in that, The steps for forming the adaptive fusion weights include: Step R1: Align the features of each stage. Mapped to state-space parameter terms : (6) (7) (8) In the formula, This is a commonly used anisotropic smooth activation function, with a step size of [missing information]. Represents the sampling interval during the discretization process; This is the weight matrix; For bias terms; It is the identity matrix; This is the discretized state transition matrix; The input projection matrix is the discretized form; Step R2 captures the correlation between levels through the state evolution equation, the formula of which is as follows: (9) Step R3: Generate dynamic fusion weights : (10) in, It is the Sigmoid activation function. The state projection matrix, The input projection matrix.
7. The zero-shot image classification method according to claim 1, characterized in that, Step S3 adopts an end-to-end training method, specifically including the following steps: Step S3-1: Construct the overall loss function for this network model: Step S3-2: Initialize the parameters of the visual encoding module, semantic encoding module, feature alignment module, multi-level feature fusion module, and classification module in the zero-shot image classification network model; Step S3-3: Input the training samples of the visible categories and the semantic information of the categories into the zero-shot image classification network model. Use the gradient descent method to iteratively update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and the trained zero-shot image classification network model is obtained.
8. The zero-shot image classification method according to claim 7, characterized in that, The overall loss function: ， in This is a semantic vector loss function based on cross-entropy loss. For based on The regression-like loss function of the paradigm These are the weight parameters of the regression loss function; Among them, the class semantic vector loss function based on cross-entropy loss ,based on Paradigm-like regression loss function They are represented as follows: ； ； in, It is an image Text information, It is an image Dynamic fusion representation of multi-stage visual features. This indicates the category to which the image belongs. This represents the total number of categories of visible images in the dataset.
9. The zero-shot image classification method according to claim 7 or 8, characterized in that, The specific steps of the gradient descent method include: Step S3-3-1: Set the initial learning rate value. Set the maximum number of iterations T, the batch size B, and initialize the network training parameters; Step S3-3-2: Divide the training samples into batches according to the batch size, and input them sequentially into the zero-sample image classification network for backpropagation to update the total loss function. The gradient, according to Forward propagation is performed in the descent direction to update the state space parameters. The model parameters are included; the above calculations are performed on each batch of input networks until the entire training set of samples has been traversed to complete one training cycle. Step S3-3-3 repeats the process of step S3-3-2 until the maximum number of iterations T is reached, resulting in a zero-shot image classification model that can automatically adjust feature contributions according to image category.
10. A zero-shot image classification device based on multi-stage modulation and dynamic fusion, used to implement the zero-shot image classification method based on multi-stage modulation and dynamic fusion as described in claim 1, characterized in that, include: The dataset acquisition module is used to acquire the dataset and divide it into training sample set and test sample set; The zero-shot image classification model is designed using an end-to-end training approach. This model includes at least: a visual encoding module, a semantic encoding module, a feature alignment module, a multi-level feature fusion module, and a classification module. in, The visual encoding module is used to perform feature modeling on the input image in at least two stages and output visual features in at least two stages. The semantic encoding module is used to encode the category text description information corresponding to the input image to obtain semantic features; The feature alignment module is used to introduce semantic information in each stage of visual feature processing, interact with the visual features and semantic features in each stage, and output multi-stage modulation and alignment features. The multi-level feature fusion module is used to construct the multi-stage alignment features into a sequence according to the stage order, and to model the sequence based on the state space model. The fusion weights of the alignment features at each stage are adaptively generated through the state evolution process to obtain the final fused alignment features. The classification module is used to output the classification result of the image based on the fused alignment features; The training module is used to train the zero-shot image classification model using training samples of the visible categories; The input / output module is used to input test samples of unseen categories into the trained zero-shot image classification network model and output the classification results of unseen categories. The feature alignment module is a dual-branch feature alignment module comprising a mapping layer, a convolutional layer, an activation function, a state space model, a normalization and residual connection layer, and a Mamba-2 module cascaded together. One branch inputs the visual features of the stage, and the other branch inputs the semantic features. The semantic information is used as a guiding signal to perform nonlinear modulation on the visual features of multiple stages.