A lightweight zero-shot image classification method and system based on a state space model

Abstract

The application provides a lightweight zero-shot image classification method and system based on a state space model, which comprises the following steps: inputting an unobserved class test sample into a trained zero-shot image classification network model to output an unobserved class image classification result; the zero-shot image classification network model comprises a visual encoder, a semantic encoder, a multimodal fusion module and a classification module; the visual encoder is used for inputting an image and outputting a visual feature; the semantic encoder is used for inputting a text and outputting a semantic feature; the multimodal fusion module is used for inputting the visual feature and the semantic feature and outputting a feature fused with the visual feature and the semantic feature; and the classification module is used for inputting the feature fused with the visual feature and the semantic feature and outputting a classification result of the image. The application has the advantages that the overall classification accuracy of the unobserved class can be maintained at a high level, and the calculation complexity of the overall model can be effectively reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a lightweight zero-shot image classification method and system based on a state space model, which can be used for planetary image perception, environmental monitoring, city planning and natural disaster management. BACKGROUND

[0002] Zero-shot learning identifies classes that have never been seen in the training phase by utilizing existing knowledge such as attribute information, text description information, etc. At present, zero-shot learning has been widely applied in terrain and landscape scene classification, plant and animal class classification, and can also be used for planetary image perception, environmental monitoring, city planning and natural disaster management. Zero-shot image classification is one of the basic tasks of zero-shot learning in the field of computer vision. It refers to identifying classes that have never been seen in the training phase under the condition that the training set and the test set are mutually exclusive, which can provide strong support for higher-level applications such as planetary image perception, environmental feature monitoring, city planning and natural disaster management.

[0003] Traditional image classification methods rely on training sets to obtain classification models, which limits the image classes that can be identified to the classes contained in the training set. However, in actual applications, the training set usually only contains a small part of the world open set, and it is difficult to include all image classes, making such methods often face problems such as data scarcity and high labeling cost. In addition, the unseen classes in actual application scenarios are usually more valuable and require high-quality recognition rates. For example, in planetary image perception, the available image data is relatively scarce, and a zero-shot classification model can acquire reasoning ability for unseen class images in the training of existing data. The perception of such new classes may bring more valuable scientific discoveries and promote people's deep exploration of knowledge related to planets. In environmental monitoring, natural or human changes may form new land cover types, and a zero-shot learning classification model can quickly adapt and accurately classify these new classes. Similarly, in disaster management, the model can efficiently identify unseen features or objects, thereby speeding up the response and decision-making speed for disasters. Therefore, it is of great significance and value to design a zero-shot image classification method that can identify unseen classes.

[0004] Most existing zero-shot image classification methods separate feature extraction and feature alignment in most cases, focusing on designing a network for feature alignment. The process of feature extraction is usually independent, that is, the image input into the pre-trained model to obtain visual features, and the text input into the natural language model to obtain semantic features. This modular design is difficult to adjust the features of vision and semantics, and it is difficult to achieve good feature alignment effect. In addition, the above feature extraction or feature alignment network is usually designed based on the Transformer backbone network, which has a high computational complexity due to its quadratic complexity, especially when modeling long sequence inputs. This makes it difficult for zero-shot image classification methods based on Transformer to be deployed on resource-limited devices, limiting their application prospects and value.

[0005] In the article "MFINet: A Novel Zero-Shot Remote Sensing Scene Classification Network Based on Multimodal Feature Interaction" (IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17:11670-11684), a zero-shot remote sensing scene classification method based on cross-modal information interaction is proposed to overcome the inconsistency of class structure between visual images and auxiliary semantics in zero-shot image classification. The method maps visual features and semantic features to latent space, and increases information interaction between visual and semantic features in latent space to fully exploit useful information of visual and semantic features and improve zero-shot remote sensing scene classification accuracy. However, this method still has the defect of not considering the computational resource constraints on actual devices, which restricts the practical application of zero-shot image classification algorithms. SUMMARY

[0006] The purpose of the present application is to overcome the above-mentioned defects, and the present application proposes a lightweight zero-shot image classification method based on state space model, comprising:

[0007] inputting the unseen class test sample into the trained zero-shot image classification network model to output the unseen class image classification result;

[0008] The zero-shot image classification network model comprises a visual encoder, a semantic encoder, a multimodal fusion module and a classification module; wherein,

[0009] The visual encoder is configured to input an image and output a visual feature;

[0010] The semantic encoder is configured to input the text and output semantic features.

[0011] The multi-modal fusion module is configured to input the visual features and the semantic features and output fused visual and semantic features.

[0012] The classification module is configured to input the fused visual and semantic features and output a classification result of the image.

[0013] The multi-modal fusion module comprises a Dual-Mamba2 block, two Mamba2 blocks and a feedforward neural network; the Dual-Mamba2 block and the two parallel Mamba2 blocks are connected in cascade, and the feedforward neural network is connected.

[0014] The Dual-Mamba2 block comprises a mapping layer, a convolution layer, an activation function, a state space model and a normalization layer.

[0015] The working process of the Dual-Mamba2 block comprises:

[0016] The input visual features are divided into a first branch and a second branch; the input semantic features are divided into a third branch and a fourth branch.

[0017] The visual features in the first branch pass through a mapping layer and an activation function to obtain a first intermediate result.

[0018] The visual features in the second branch pass through a mapping layer and an activation function to obtain a second intermediate result; the second intermediate result comprises auxiliary parameters comprising image features and second intermediate visual features; the second intermediate visual features of the second intermediate result are replaced by third intermediate semantic features of a third intermediate result to input a state space model; an output result of the state space model is connected with the first intermediate result, and then output image semantic features after passing through a normalization layer and a mapping layer.

[0019] The semantic features in the third branch pass through a mapping layer and an activation function to obtain a third intermediate result; the third intermediate result comprises auxiliary parameters comprising text features and third intermediate semantic features; the third intermediate semantic features of the third intermediate result are replaced by second intermediate visual features of a second intermediate result to input a state space model; an output result of the state space model is connected with the fourth intermediate result, and then output text visual features after passing through a normalization layer and a mapping layer.

[0020] The semantic features in the fourth branch pass through a mapping layer and an activation function to obtain a fourth intermediate result.

[0021] As an improvement of the above method, the visual encoder is a VMamba.

[0022] As an improvement to the above method, the semantic encoder is Mamba.

[0023] As an improvement to the above method, the loss function L of the zero-shot image classification network model zsl for:

[0024] L zsl =L c +λ·L r

[0025] Among them, L r For the regression-like loss function based on L2 paradigm:

[0026]

[0027] Where y represents the text information of the image; The output representation of the image; ||·||2 represents the L2 normal form;

[0028] L c The class semantic vector loss function based on cross-entropy loss is as follows:

[0029]

[0030] in, The category to which the image belongs is represented; C S This represents the total number of categories of visible images in the dataset.

[0031] λ is the weight parameter of the regression loss function.

[0032] As an improvement to the above method, the zero-shot image classification network model is trained end-to-end, using gradient descent to iteratively update the parameters in the zero-shot image classification network model, as follows:

[0033] 1) Set the initial learning rate, maximum number of iterations, and batch size to initialize the network training parameters;

[0034] 2) Divide the training samples into batches according to the batch size, and input them into the zero-sample image classification network model in turn to backpropagate and update the gradient of the total loss function. Then, perform forward propagation to update the network training parameters according to the direction of gradient descent. Perform the above calculations for each batch input into the network until the entire training set of samples has been traversed to input into the network to complete one training cycle.

[0035] 3) Repeat step 2) until the maximum number of iterations is reached, and a trained zero-shot image classification model is obtained.

[0036] This application also provides a lightweight zero-shot image classification system based on a state-space model, implemented using the above method, the system comprising:

[0037] The sample classification module is used to input unseen class test samples into a trained zero-shot image classification network model and output the classification results of unseen class images.

[0038] The model training module is used to train the zero-shot image classification network model.

[0039] Compared with existing technologies, the advantages of this application are:

[0040] 1. This invention constructs a zero-shot image classification network that includes a visual encoder, a semantic encoder, a multi-modal fusion module, and a classifier. It integrates the feature extraction network, the feature alignment network, and the classifier into a unified framework, allowing the input data to directly obtain the output result through the trained zero-shot image classification network. This avoids manual preprocessing and subsequent processing, and improves the zero-shot classification speed of images.

[0041] 2. The Dual-Mamba2 module based on Mamba2 designed in this invention can effectively extract the interaction information between the two modalities by exchanging image and text parameters.

[0042] 3. The multi-modal fusion module based on Dual-Mamba2 designed in this invention effectively realizes feature interaction between visual and semantic modalities through the collaborative cascading configuration of Dual-Mamba2 module and Mamba2 block.

[0043] In summary, compared with existing technologies, this invention can better utilize the shared information between images and auxiliary information. Moreover, under the zero-shot setting, it can not only maintain a high level of overall classification accuracy for unseen categories, but also effectively reduce the computational complexity of the model and reduce the computing power requirements of the platform on which the model is mounted. This is more conducive to the deployment of zero-shot image classification methods on hardware devices and can provide better services for higher-level applications such as ground feature monitoring and change detection. Attached Figure Description

[0044] Figure 1 The diagram shows a flowchart of a lightweight zero-shot image classification method based on a state-space model.

[0045] Figure 2 The image shows a schematic diagram of the Dual-Mamba2 module;

[0046] Figure 3 The diagram shown is a block diagram of a zero-shot image classification model.

[0047] Figure 4 The image shown is an example of a self-constructed Mars scene image category. Detailed Implementation

[0048] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0049] This application proposes a lightweight zero-shot image classification method and system based on a state space model (SSM) to fully utilize the extracted visual and semantic information, enhance the information interaction between visual and semantic features, reduce the computational complexity of the model, and improve the zero-shot image classification performance.

[0050] The technical approach to achieve the above objectives is to design visual semantic feature extraction and zero-shot classification as a unified process. Visual and semantic encoders based on state-space models are used to mine the input image features and text information respectively. A feature alignment network based on state-space models is used to achieve interactive learning between visual and semantic features, making the acquired visual-semantic features more discriminative, obtaining more accurate feature representation, effectively reducing the computational complexity of the model, and ensuring classification accuracy.

[0051] Example 1

[0052] like Figure 1 As shown, the lightweight zero-shot image classification method based on a state-space model provided by this invention includes the following steps:

[0053] 1. For example Figure 3 As shown, an end-to-end training approach is adopted to design a zero-shot image classification model consisting of four modules: a visual encoder (Visual StateSpace Model, VMamba), a semantic encoder (Mamba), a dual-stream Mamba Feature Fusion Module (DM2F) based on Dual-Mamba2 units, and a classification module.

[0054] 1) Build a visual encoder based on a state-space model.

[0055] Image features are extracted using a bidirectional scanning method, and a VMamba-based visual encoder is built.

[0056] 2) Build a semantic encoder based on a state-space model.

[0057] We use a selective scanning mechanism to extract semantic features from text and build a state-space model-based semantic encoder called Mamba.

[0058] 3) Build the Dual-Mamba2 module.

[0059] like Figure 2As shown, the Dual-Mamba2 module is a two-branch network, where each branch consists of a cascaded input mapping layer, a one-dimensional convolutional layer, an activation function (SiLU), a state-space model, a normalization layer, and an output mapping layer.

[0060] First, image and text features are processed separately on two mapping layers to obtain auxiliary parameters A that include image features. v B v and C v Auxiliary parameter A containing text features t B t and C t The specific formula is as follows:

[0061]

[0062] in, The formula for calculating the dual layer of the state space.

[0063] The output consists of two branches, each divided into two information streams. One information stream is processed by a convolutional layer and then enters the core state space model through an activation function. The specific formula is as follows:

[0064]

[0065]

[0066] When entering the state-space model, swap X v and X t The input location is passed to the state space model for calculation.

[0067] The other information stream only passes through the activation function, the specific formula of which is as follows:

[0068]

[0069] The two are then concatenated, and after passing through a normalization layer and an output mapping layer, the final output is obtained, as shown in the following formula:

[0070]

[0071] The output sequence Z has two forms: the visual features Z corresponding to the text. v The semantic features Z corresponding to the image t ; The output mapping matrix, Let Ω(·) be the bias matrix of the output, and let Ω(·) denote the normalization function. The normalized output further refines the fused features, ensuring robust and consistent multimodal alignment.

[0072] 4) Construct the multi-mode fusion module DM2F.

[0073] This application introduces a novel hierarchical structure that cascades a Dual-Mamba2 module with two Mamba2 blocks, as shown in the following formula:

[0074] Y v ′=Γ(A v ′,Z v B v ′,C v ′)

[0075] Y t ′=Γ(A t ′,Z t B t ′,C t ′)

[0076] Where Γ(·) is the formula for calculating Mamba2, A v ′、B v ′、C v ′ and A t ′、B t ′、C t ′ are auxiliary parameters for the image and text, respectively, Z v and Z t These are the interaction features obtained by the Dual-Mamba2 module.

[0077] 2. Train the zero-shot image classification model:

[0078] 1) Construct the overall loss function L for this network model. zsl =L c +λ·L r L c L is a class semantic vector loss function based on cross-entropy loss. r Let λ be the class regression loss function based on the L2 paradigm, and λ be the weight parameter of the class regression loss function.

[0079] The semantic vector loss function L based on cross-entropy loss c L2-based regression loss function L r They are represented as follows:

[0080]

[0081] Where y represents the text information of image x. It is the output representation of image x. C represents the category to which an image belongs. S This represents the total number of visible image categories in the dataset; ||·||2 represents the L2 paradigm.

[0082] 2) The parameters for initializing the zero-shot image classification network model include auxiliary parameters A1, X1, B1, and C1 in the visual encoder VMamba, auxiliary parameters A2, X2, B2, and C2 in the semantic encoder Mamba, and auxiliary parameter A in the multimodal fusion module DM2F. v X v B v and C v A t X t B t and C t ;

[0083] 3) Input the visible category training samples and category semantic information into the zero-shot image classification network model, and use the gradient descent method to iteratively update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and the trained zero-shot image classification network model is obtained.

[0084] Specifically, the gradient descent method is used to iteratively update the parameters in the zero-shot image classification network, as follows:

[0085] a) Set the initial learning rate α, the maximum number of iterations T, the batch size B, and initialize the network training parameters;

[0086] b) Divide the training samples into batches according to batch size, and input them sequentially into the zero-sample image classification network for backpropagation to update the total loss function. The gradient, according to The network training parameters are updated by forward propagation in the descent direction. This calculation is performed on each batch of input network samples until the entire training set has been traversed, completing one training cycle.

[0087] c) Repeat process b) until the maximum number of iterations T is reached, and a trained zero-shot image classification model is obtained.

[0088] 3. Input the unseen class test samples into the trained zero-shot image classification network model, and output the classification results of the unseen class images.

[0089] Example 2

[0090] This embodiment uses the ZS-Mars dataset, a standard dataset for classifying Mars scenes with zero samples, as an example to classify it. The images in this dataset and their corresponding ground truth labels are as follows: Figure 4 As shown, it includes 10 types of scenarios.

[0091] Reference Figure 1 The specific implementation steps of this example are as follows:

[0092] Step 1: Randomly divide the training and test samples.

[0093] 1.1) For the ZS-Mars dataset, the training (visible class) and test (unseen class) samples are divided into training (visible class) and test (unseen class) samples according to the ratio of 6 / 4, 7 / 3, and 8 / 2, that is, 6 visible class / 4 unseen class, 7 visible class / 3 unseen class, and 8 visible class / 2 unseen class;

[0094] 1.2) Encode the label of each training sample into a vector y in one-hot encoding. i Each test sample's label is encoded in one-hot format as a vector q. i .

[0095] Step 2: Build a visual encoder based on a state-space model.

[0096] 2.1) Image features are extracted using a bidirectional scanning method. A visual encoder VMamba based on Mamba is built, with an input image size of 224×224×3.

[0097] 2.2) From Figure 3 The samples shown are divided into training and test sets and their corresponding labels according to step 1. The training set is represented as X = {x1, ..., x...}. train}, with labels Y = {y1,...,y train}, x i For any training sample, for different visible / invisible category ratios, the subscript 'train' corresponds to the product of the number of visible categories and the number of scenes in each category, x i The corresponding label y i ∈{1,2,...,6}; the test set is represented as P={p1,...,p test}, with labels Q = {q1,...,q} test}, p i For any test sample, for different visible / invisible category ratios, the subscript 'test' corresponds to the product of the number of invisible categories and the number of scenes in each category, p i The corresponding tag q i ∈{1,2,3,4};

[0098] Step 3: Build a semantic encoder based on a state-space model.

[0099] 3.1) Extract semantic features from the text using a selective scanning mechanism and build a semantic encoder Mamba based on a state space model.

[0100] from Figure 3The sample labels shown are obtained by dividing the training set and the test set according to step 1. Specifically, for different visible / invisible category ratios, the number of training samples corresponds to the product of the number of visible categories and the number of scenes in each category, while the number of test samples corresponds to the product of the number of invisible categories and the number of scenes in each category. The remaining sample labels are used as the test set.

[0101] Step 4: Construct a multi-mode fusion module based on a state-space model.

[0102] Reference Figure 2 The specific implementation of this step is as follows:

[0103] 4.1) Configure the Dual-Mamba2 module, which consists of mapping layers, convolutional layers, activation functions, a state-space model, and normalization layers. The specific computation flow is as follows:

[0104] First, establish a mapping layer. Set up four mapping modules All are composed of linear matrices, and the specific formulas are as follows:

[0105]

[0106] The input sequence X has two types: visual feature X. v and semantic features X t W v in W t in Let b be the corresponding input mapping matrix. v in b t in This is the corresponding input bias matrix.

[0107] The output contains two branches, each branch consisting of two information streams. One information stream is processed by a convolutional layer, then passes through an activation function before entering the core state-space model module. Upon entering the state-space model, X is swapped. v and X t The input location, and the specific formula are as follows:

[0108]

[0109] The other information stream only passes through the activation function, the specific formula of which is as follows:

[0110]

[0111] The two are then concatenated, and after passing through a normalization layer and an output mapping layer, the final output is obtained, as shown in the following formula:

[0112]

[0113] The output sequence Z has two forms: the visual features Z corresponding to the text. v The semantic features Z corresponding to the image t W v out W t out The output mapping matrix, Let Ω(·) be the bias matrix of the output, and let Ω(·) denote the normalization function.

[0114] 4.2) Configure the multi-mode fusion module, which consists of a Dual-Mamba2 module and two parallel Mamba2 blocks.

[0115] First, we create the Dual-Mamba2 module to obtain the visual features Z corresponding to the text. v The semantic features Z corresponding to the image t The specific formula is as described in 4.1).

[0116] Subsequently, the aforementioned features are input into parallel Mamba2 blocks, whose structure is similar to that of the Dual-Mamba2 module, containing mapping layers, convolutional layers, activation functions, state-space models, and normalization layers. The computation process of the parallel Mamba2 blocks is the same, and its specific computation flow is as follows:

[0117] Establish a mapping layer Set up four mapping modules All are composed of linear matrices, and the specific formulas are as follows:

[0118] Z′=W in ·Z+b in

[0119] Where Z is the input sequence, i.e., the visual feature Z. v Or semantic feature Z t W in Let b be the input mapping matrix. in This is the input bias matrix.

[0120] The output consists of two information streams. One stream is processed by a convolutional layer and then passes through an activation function before entering the core state-space model module. The specific formula is as follows:

[0121] Z c =Conv(Z′)

[0122] Z ssm =SSM(SiLU(Z) c ))

[0123] The other information stream only passes through the activation function, the specific formula of which is as follows:

[0124] Z a =SiLU(Z′)

[0125] The two are then concatenated, and after passing through a normalization layer and an output mapping layer, the final output is obtained, as shown in the following formula:

[0126] Z out =Z ssm +Z a

[0127] Y = W out ·Ω(Z out )+b out

[0128] Where Y is the output sequence, i.e., visual feature Y v Or semantic feature Y t W out Let b be the output mapping matrix. out Let Ω(·) be the bias matrix of the output, and let Ω(·) denote the normalization function.

[0129] Finally, the outputs of the Dual-Mamba2 module and the parallel Mamba2 block are concatenated and passed through a feedforward neural network layer to obtain the final output, as shown in the following formula:

[0130] Y v =FNN(Z) v +Y v )

[0131] Y t =FNN(Z) t +Y t )

[0132] Where FNN(·) represents the calculation formula for the feedforward neural network layer.

[0133] Step 5: Construct a zero-shot image classification model based on a state-space model.

[0134] 5.1) The above-mentioned visual encoder, semantic encoder, and multimodal fusion module are cascaded in sequence to form a visual semantic feature extraction and alignment encoder;

[0135] 5.2) Constructing the classifier g η g η It includes one fully connected layer that transforms the size of the transformation matrix to align the visual semantic features extracted above with the encoder output features, transforming them into probability vectors belonging to each category. Used to determine the category to which a sample belongs.

[0136] Step 6: Construct the loss function for the zero-shot image classification model based on the state-space model.

[0137] 6.1) Constructing a regression-like loss function L based on the L2 paradigm r This is used to constrain the image representation to ensure that it is accurately mapped to the corresponding text embedding. The specific formula is as follows:

[0138]

[0139] Where x represents the visual features of the image, and y represents the textual information of the image;

[0140] 6.2) Construct the cross-entropy loss function L c This is used to force an image to have the highest compatibility score with its corresponding class semantic vector. The specific formula is as follows:

[0141]

[0142] in, C represents the category to which an image belongs. S This represents the total number of categories of visible images in the dataset;

[0143] Step 7: Train the parameters of the zero-shot image classification model using training samples and the gradient descent method.

[0144] 7.1) Set the initial learning rate α, the maximum number of iterations T, and the batch size B;

[0145] 7.2) The parameters for initializing the zero-shot image classification network model include auxiliary parameters A1, X1, B1, and C1 in the visual encoder VMamba, auxiliary parameters A2, X2, B2, and C2 in the semantic encoder Mamba, and auxiliary parameter A in the multimodal fusion module DM2F. v X v B v and C v A t X t B t and C t , and the parameter η in the classifier;

[0146] 7.3) Divide the training samples into batches according to the batch size, and input them sequentially into the unbalanced hyperspectral image classification network for backpropagation to update the total loss function. The gradient, according to The network training parameters are updated by forward propagation in the descent direction. The above calculation is performed on each batch of input network until the entire training set samples are input network to complete one training cycle.

[0147] 7.4) Repeat the process in 7.3) until the maximum number of iterations T is reached, and a trained zero-shot classification model is obtained.

[0148] Step 8: Use the trained zero-shot image classification model to classify the test samples.

[0149] Each test sample is input into the trained zero-shot image classification model to obtain the probability p(y|x) of the label to which each test sample belongs:

[0150]

[0151] Where softmax is the normalization function, L zsl Let x be the total loss function, and y be the test sample. c Let z be the label corresponding to the c-th class of test samples. c Let A1, X1, B1, and C1 be the semantic features corresponding to the c-th class of test samples, C be the total number of test samples, A1, X1, B1, and C1 be the auxiliary parameters in the visual encoder VMamba, and A2, X2, B2, and C2 be the auxiliary parameters in the semantic encoder Mamba. v X v B v and C v A t X t B t and C t η represents the auxiliary parameters of the multi-modal fusion module DM2F, and η represents the trainable parameters of the classifier.

[0152] The effects of this invention can be further illustrated by the following simulation results.

[0153] Test data:

[0154] Using ZS-Mars data, semantic information is expressed as sentence vectors representing long sequence inputs. Images are tested according to three visible / invisible ratios: a 6 / 4 ratio (6 visible images for training, 4 unseen images for testing); a 7 / 3 ratio (7 visible images for training, 3 unseen images for testing); and an 8 / 2 ratio (8 visible images for training, 2 unseen images for testing). The specific visible / invisible category divisions are shown in Table 1.

[0155] Test environment:

[0156] It uses a Linux system, an NVIDIA GeForce GTX 4090 GPU, and the PyTorch deep learning framework.

[0157] Table 1 Classification of Visible / Invisible Categories

[0158]

[0159]

[0160] Simulation content:

[0161] Simulation 1: Under the above conditions, the existing zero-shot image scene classification method MFINet was used to classify ZS-Mars data. The results are shown in Table 2.

[0162] Simulation 2: Under the above conditions, the ZS-Mars data were classified using the method of this invention, and the results are shown in Table 2.

[0163] The comparison in Table 2 shows that the method proposed in this invention achieves higher classification accuracy with lower computational complexity, thus realizing better classification performance.

[0164] Evaluation indicators:

[0165] The two methods were tested 10 times using the same random number, and the overall classification accuracy OA was calculated. The test results are shown in Table 2.

[0166] The formula for calculating the indicator is as follows:

[0167]

[0168] Where TP is the number of samples correctly classified into this class, FN is the number of samples in this class that were incorrectly classified into other classes, TN is the number of samples in other classes that were not classified into this class, and FP is the number of other samples that were incorrectly classified into this class.

[0169] Table 2 Simulation results of the present invention and the comparative method

[0170]

[0171] As can be seen from Table 2, compared with the current advanced MFINet method, the present invention can achieve higher classification accuracy with less resource consumption under three visible / invisible ratios.

[0172] Example 3

[0173] This application also provides a lightweight zero-shot image classification system based on a state-space model, implemented using the above method, the system comprising:

[0174] The sample classification module is used to input unseen class test samples into a trained zero-shot image classification network model and output the classification results of unseen class images.

[0175] The model training module is used to train the zero-shot image classification network model.

[0176] This application may also provide a computer device, including: at least one processor, memory, at least one network interface, and a user interface. The various components in this device are coupled together via a bus system. It is understood that the bus system is used to implement communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0177] The user interface can include a display, keyboard, or clicking device. Examples include a mouse, trackball, touchpad, or touchscreen.

[0178] It is understood that the memory in the embodiments disclosed in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0179] In some implementations, the memory stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0180] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. Programs implementing the methods of the embodiments of this disclosure can be included in the application programs.

[0181] In the above embodiments, the processor can also invoke programs or instructions stored in memory, specifically programs or instructions stored in an application program, for the following purposes:

[0182] Follow the steps described above.

[0183] The above methods can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed above. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the disclosed methods can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0184] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0185] For software implementation, the technology of this application can be implemented by executing the functional modules (e.g., procedures, functions, etc.) of this application. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or externally.

[0186] This application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, it can implement the steps in the above method embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A lightweight zero-shot image classification method based on a state-space model, comprising: Input the unseen class test samples into the trained zero-shot image classification network model, and output the classification results of the unseen class images; The zero-shot image classification network model includes a visual encoder, a semantic encoder, a multimodal fusion module, and a classification module; wherein, The visual encoder is used to input an image and output visual features; The semantic encoder is used to input text and output semantic features; The multimodal fusion module is used to input visual features and semantic features, and output features that fuse visual and semantic features; The classification module is used to take as input visual and semantic features and output the classification result of the image. The multi-mode fusion module includes a Dual-Mamba2 block, two Mamba2 blocks, and a feedforward neural network; the Dual-Mamba2 block and the two parallel Mamba2 blocks are cascaded and then connected to the feedforward neural network. The Dual-Mamba2 block includes a mapping layer, a convolutional layer, an activation function, a state-space model, and a normalization layer; The working process of the Dual-Mamba2 block includes: The input visual features are divided into the first and second branches; the input semantic features are divided into the third and fourth branches. In the first branch, the visual features pass through a mapping layer and an activation function to obtain the first intermediate result; In the second branch, the visual features pass through a mapping layer and an activation function to obtain a second intermediate result. The second intermediate result includes auxiliary parameters containing image features and a second intermediate visual feature. The second intermediate visual feature of the second intermediate result is replaced with the third intermediate semantic feature of the third intermediate result and then input into the state space model. The output of the state space model is concatenated with the first intermediate result and then passed through a normalization layer and a mapping layer to output the semantic features of the image. The semantic features in the third branch pass through a mapping layer and an activation function to obtain the third intermediate result; the third intermediate result includes auxiliary parameters containing text features and the third intermediate semantic features; the third intermediate semantic features of the third intermediate result are replaced with the second intermediate visual features of the second intermediate result and then input into the state space model; the output of the state space model is connected with the fourth intermediate result and then passed through a normalization layer and a mapping layer to output the visual features of the text. The semantic features in the fourth branch pass through a mapping layer and an activation function to obtain the fourth intermediate result.

2. The lightweight zero-shot image classification method based on a state-space model according to claim 1, characterized in that: The visual encoder is VMamba.

3. The lightweight zero-shot image classification method based on a state-space model according to claim 1, characterized in that: The semantic encoder is Mamba.

4. The lightweight zero-shot image classification method based on a state-space model according to claim 1, characterized in that: The loss function of the zero-shot image classification network model for: ； in, For based on The regression-like loss function of the paradigm: ； in, Text information for the image; The output representation of the image; express Paradigm; The class semantic vector loss function based on cross-entropy loss is as follows: ； in, This represents the category to which the image belongs; This represents the total number of categories of visible images in the dataset. These are the weight parameters of the regression loss function.

5. The lightweight zero-shot image classification method based on a state-space model according to claim 1, characterized in that: The zero-shot image classification network model is trained end-to-end, using gradient descent to iteratively update the parameters of the zero-shot image classification network model, as follows: 1) Set the initial learning rate, maximum number of iterations, and batch size to initialize the network training parameters; 2) Divide the training samples into multiple batches according to the batch size, perform backpropagation to update the gradient of the total loss function for each batch of zero-sample image classification network model, and perform forward propagation to update the network training parameters according to the gradient descent direction. 3) Repeat step 2) until the maximum number of iterations is reached, and a trained zero-shot image classification model is obtained.

6. A lightweight zero-shot image classification system based on a state-space model, implemented according to the method of any one of claims 1-5, characterized in that, The system includes: The sample classification module is used to input unseen class test samples into a pre-trained zero-shot image classification network model and output the classification result of the unseen class image; and The model training module is used to train the zero-shot image classification network model.

Images